Combining Adaptive Survey Design with Post-Survey Weighting Adjustment

Abstract

Adaptive survey design is a recruitment method that tailors survey protocols to distinctive characteristics of sample subgroups. The goal of adaptive design is to improve response rates and balance respondent composition, which may in turn lead to survey estimates with lower biases and variances. However, it is not well understood how adaptive design can improve survey estimates beyond the standard practice of post-survey weighting adjustment. This question is asked because, just like adaptive design, the purpose of weighting adjustment is also to reduce the biases and variances of survey estimates. In addition, the input information available for developing adaptive design is also available for performing weighting adjustment. So can combining adaptive design and weighting adjustment lead to survey estimates with small biases and variances compared to only performing weighting adjustment?

This dissertation investigates this question with three empirical studies. The first is a simulation study that incorporated practical constraints into theorizing the additional effect of adaptive design. The simulation suggested that practical constraints could substantially curb the benefits of adaptive design. When the adaptive design was developed based on inadequate input information, it backfired and led to estimates with larger biases, though its benefit of reducing variances was retained.

The second study reports the results of an experiment that evaluated the additional effect of adaptive design in the Detroit Metro Area Communities Study. The adaptive design was evaluated in five dimensions: 1) response rates, 2) demographic composition of respondents, 3) bias and variance of key survey estimates, 4) changes in significant results of regression models, and 5) costs. The most significant benefit of the adaptive design was its ability to generate more efficient survey estimates with smaller variances and smaller design effects.

The third study extends the investigation to a national context and investigates adaptive design in a two-stage national probability push-to-web/mail survey, the American Family Health Study. The adaptive design did not change survey estimates above and beyond the weighting adjustment. The experiment yields useful insights into the implementation of adaptive design in a self-administered mail-recruit web/mail survey.