Predicting and Preventing Breakoff in Web Surveys

Abstract

Due to recent general shifts from mail to the web in survey data collection modes, respondents who break off prior to completing web surveys have become a more prevalent problem in data collection. Given the already lower response rates in web surveys as compared to more traditional modes, such as face-to-face interviewing, it is crucial to keep as many diverse respondents in a web survey as possible. This action will help prevent breakoff bias, and thus maintain high data quality and produce more accurate survey estimates. To prevent and reduce web survey breakoffs, Chapter 4 of this dissertation aimed to understand the breakoff process and its associated variables. The typical breakoff respondent: tended to be female; was non-white; was a student; waited for email reminders to start the questionnaire; and answered on a mobile device. Respondents who had broken off the questionnaire in previous waves were more likely to quit the questionnaire again very early on.

Based on the findings from Chapter 4, predictions were then made about breakoff timing at the page-level in the second paper. In addition to well-known factors associated with breakoff, such as using a mobile device, Chapter 5 examined the relationships of previous response behaviors like speeding and item nonresponse with breakoff timing. This allowed for predictions about the risk of quitting for each respondent at the page-level using Cox survival models. Male respondents tended to quit at the beginning of the questionnaire, while female respondents had a higher risk of quitting toward the end of the questionnaire. There was no significant difference in breakoff risks between mobile respondents and non-mobile respondents at the beginning of the questionnaire. This quickly changed with every page completed by mobile respondents. Item nonresponse and extensive scrolling behavior were both positively associated with the risk of breaking off. Short response times and response time changes (speeding up and slowing down) both increased the risk of quitting the questionnaire.

Finally, in a real-time experiment implemented for Chapter 6, interventions were conducted with respondents who had a high predicted probability of breaking off from a web survey. For this approach, a prediction model was implemented in the next wave of a panel study, and this model evaluated the risk of breaking off on every page for each respondent in addition to comparing the estimated risk with an established threshold. If the estimated risk exceeded the threshold, then the respondents saw a motivational pop-up message reminding them of their commitment to completing the questionnaire. Females, students, Blacks, and respondents on mobile devices reacted positively when assigned to the treatment group and showed less undesirable response behavior than respondents in the control group.

The dissertation concludes with recommendations for practice and suggested directions for future work in this area.