Mobile Web Surveys: a First Look at Measurement, Nonresponse, and Coverage Errors.

Abstract

This dissertation focuses on the use of smartphones for Web surveys. The current state of knowledge about whether respondents are willing and able to accurately record their answers when using such devices is evolving, but far from complete. The primary purpose of my research is therefore to investigate the implications of this new mode for various sources of error using a Total Survey Error (TSE) perspective.

Each chapter reports on a different aspect of a mode experiment that I designed to compare the effect of completion device (smartphone vs. computer) on survey errors. The experiment was carried out using the LISS panel (Longitudinal Internet Studies for the Social Sciences), a probability-based Web panel administered by CentERdata at Tilburg University in the Netherlands.

The first analysis (Chapter 2) compares response quality in the two modes. When using smartphones, respondents in this study really were more mobile and more engaged with the other people and other tasks compared to when using computers. Despite this, response quality – conscientious responding and disclosure of sensitive information – was equivalent between the two modes of data collection.

The second analysis (Chapter 3) investigates the causes of nonresponse in the mobile Web version of the experiment. I found that several social, psychological, attitudinal, and behavioral measures are associated with nonresponse. These include factors known to influence participation decisions in other survey modes such as personality traits, civic engagement, and attitudes about surveys as well as factors that may be specific to this mode, including smartphone use, social media use, and smartphone e-mail use.

The third analysis (Chapter 4) estimates multiple sources of error simultaneously in the mobile Web version of the experiment. Errors are estimated as a mode effect against the conventional Web survey, which serves as the benchmark. I find few overall mode effects and no evidence whatsoever of measurement effects, but a significant impact of non-coverage bias for over one-third of the estimates.

Collectively, these findings suggest that non-observation errors (i.e., coverage and nonresponse), not measurement errors, are the largest obstacle to the adoption of mobile Web surveys for population-based inference.