The Effects of Respondent Commitment and Feedback on Response Quality in Online Surveys

Abstract

Answering questions completely, accurately and honestly is not always the top priority for survey respondents. In their pioneering work in the 1970s and 80s, Charles Cannell and colleagues at the University of Michigan demonstrated the promise of directly asking respondents to commit to providing complete and accurate answers. While promising, these studies were conducted decades ago, in interviewer administered modes, with limited data quality measures. This dissertation consists of two experimental studies investigating the effectiveness of commitment as well as automatic feedback in improving data quality in online surveys. Study 1 measures the effect of commitment – “yes” or “no” – in an online labor force survey. Study 2 measures the effect of asking respondents to commit to engaging in several response behaviors that seem likely to promote data quality, such as reading the questions carefully, and trying to be as precise as possible, in an survey of parents about their child’s health and healthcare. Study 2 also examines the effect of providing feedback in response to behaviors that are associated with reduced data quality such as speeding and reporting an incomplete date. Both studies verify the accuracy of select responses using administrative records, in contrast to the indirect quality measures in previous evaluations of commitment and feedback. In Study 1, results were stronger for those who committed versus those who were invited to commit but did not, and in Study 2 for those who committed to all of the requested response behaviors versus those who committed to engage in a few. Study 2 also found that feedback substantially reduced unwanted respondent behaviors and improved data quality by some measures over and above the effect of commitment. Overall, in both studies, commitment had a positive effect on data quality including response accuracy, much as Cannell and his colleagues would have expected, although in Study 2, some negative consequences were also evident. For example, it appeared that committed respondents were more likely to skip a question rather than answer it inadequately. On balance, practitioners will likely find the trade-offs produced by asking for commitment to be favorable and worth considering in production surveys.