The Error Properties of Interviewer Observations and their Implications for Nonresponse Adjustment of Survey Estimates.

Abstract

Interviewer observations are an important source of auxiliary information in survey research. Interviewers can record observations for all units in a sample, and selected observations may be associated with both key survey variables and response propensity. Survey statisticians use auxiliary variables with these properties to compute post-survey nonresponse adjustments to survey estimates that reduce both bias and variance in the estimates engendered by nonresponse. Unfortunately, interviewer observations are typically judgments and estimates, making them prone to error. To date, no studies have considered the implications of these errors for the effectiveness of nonresponse adjustments, effective observational strategies leading to reduced error rates, predictors of observation accuracy in face-to-face surveys, or alternative estimation methods for mitigating the effects of the errors on estimates. This dissertation presents results from three research studies designed to fill these important gaps in the existing literature.

The first study 1) analyzes the error properties of two interviewer observations collected in the National Survey of Family Growth (NSFG), finding accuracy rates ranging from 72-78% and evidence of systematic errors; 2) examines the effectiveness of nonresponse adjustments based in part on the observations, finding evidence of associations with key NSFG variables and response propensity but only slight shifts in estimates; and 3) simulates the implications of errors in the observations for the effectiveness of weighting class adjustments for nonresponse, finding that adjustments based on the error-prone observations attenuate possible reductions in bias. The second study uses multilevel modeling techniques to identify several respondent- and interviewer-level predictors of accuracy in the two NSFG observations, including those supported by social psychological theories of what leads to improved judgment accuracy. The third study develops pattern-mixture model (PMM) estimators of means for the case when an auxiliary variable is error-prone, true values for the variable are collected from survey respondents, and the true values are predictive of unit nonresponse under a non-ignorable missing data mechanism. Simulation studies show that the PMM estimators have several favorable properties in these situations relative to other popular estimators, and R code is provided implementing the PMM approaches.