網路調查的注意力檢測對題項回應時間與問卷完填的影響

The Effects of Attention Checks on Item Response Latency and Questionnaire Completion

近二十年來，網路調查有益趨流行的趨勢。由於網路調查仰賴受訪者自填問卷，許多網路調查會在問卷中加入注意力檢測題，以偵測受訪者的專注程度。儘管注意力檢測題已廣被應用，嚴謹的方法研究仍不多見。特別是注意力檢測題對受訪者後續作答行為、退出問卷機率的影響，研究極少。在這項研究中，針對機率樣本進行網路調查實驗，設計了兩個注意力檢測題，並區分前、後兩個放置位置，再加上未設有注意力檢測題的對照組，總共區分為五組。藉由蒐集到的問卷資料、周邊資料，分析注意力檢測題對退出問卷機率、題項回應時間的影響效果。分析結果顯示，注意力檢測題對退出問卷機率並無顯著的影響效果。但在題項回應時間方面，放置於問卷後半部的注意力檢測題，會使得後續題項的回應時間顯著增加；放置於前半部的注意力檢測題，則沒有顯著影響。這些結果顯示，如果研究者想在網路調查中藉由注意力檢測題提升受訪者的專注力，將注意力檢測題放在問卷後半部較前半部來得有效。

Online surveys have become more prevalent due to the widespread use of the Internet and mobile devices. Because online survey data are collected by self-administered questionnaires, data quality is highly related to respondents’ attentiveness. Attention checks have been developed by researchers to identify inattentive respondents in online surveys by means of asking the respondents to respond according to the instructions of the question rather than their true answer.
Even though passing an attention check was found to be associated with the respondent’s overall data quality, several researchers have argued that a respondent may feel uncomfortable or not trusted when he or she encounters an attention check question. However, only a few studies, mostly based on non-probability samples, have examined whether attention checks affect respondents’ subsequent behaviors. By conducting a survey experiment centered on a probability-based online panel, this study analyzes the effects of attention checks on the likelihood of dropping out from an online survey and the response latency to subsequent items.
The sample used in this study is based on an online panel constructed by the Center for Survey Research, Academia Sinica, Taiwan. To examine whether the design and location of attention check questions matter for the respondents’ subsequent behaviors, we adopted a two-by-two experimental design in which there are two designs and two locations (former half and latter half) for attention check questions. This sample was randomly assigned to these four treatment groups and one additional control group (with no attention check questions).
The theme of the online survey is the attitudes toward surveys and data. The online survey was conducted from April 19 to May 3, 2021. Invitation emails were sent successfully to 7,400 subjects. Among the 3,875 respondents, 3,770 completed the questionnaires. A logit model and Cox proportional hazard model were used to analyze the effects of attention checks on the probability of withdrawal. A multilevel mixedeffects generalized linear model was applied to estimate the impact of an attention check on response latency of subsequent items. Possible outliers of response latency were identified and treated in different ways as a robustness check.
The results indicate that adding an attention check to a questionnaire does not matter for the probability of withdrawal. This finding is intact with regards to the design or location of attention checks. Our findings also reveal that attention checks located in the latter half of a questionnaire have positively significant effects on the response latency of subsequent items, yet the attention checks placed in the former half are insignificant. The findings on response latency are robust to different ways of identifying and treating outliers. In sum, our results indicate that even though attention checks are not associated with the probability of completing a questionnaire, their locations are important for whether respondents’ attentiveness can be effectively elevated.
Despite our finding that attention checks did not provoke negative behaviors such as quitting the questionnaire or speeding up the response time, a preliminary analysis on response latency to attention checks reveals that the respondents who failed an attention check spent more time in responding to the attention check than those who passed. Responding behaviors to the attention check itself deserve further exploration in future research.