Answerable questions: Advances in the methods for identifying and resolving questionnaire problems in survey research.

Abstract

If survey questions are to generate valid data, respondents must be able to understand the material within, able and motivated to provide the information requested, and able to express their responses in the format provided. Three studies are presented, each focusing on one of these evaluative criteria for survey questions. Each study also considers strengths and weaknesses of various methods used to evaluate questionnaires. The first study examined questions with characteristics that might make them difficult to understand: complex question structure, unwarranted presuppositions, or confusing levels of detail. Alternative question versions that varied the structure, or eliminated other potential problems, were administered via split ballot in a national survey. Response distributions and rates of various interviewer and respondent behaviors suggested that eliminating presuppositions has an important impact on response distributions and ease of administration. Evidence also emerged that it is preferable to keep definitions and explanations simple. No evidence emerged to contradict conventional guidelines for structuring questions. The second study explored how knowledge, interpretations of researcher expectations, and motivation affect the decision of whether or not to answer a survey question. Respondents answered a short questionnaire with typical survey items, and then answered follow-up questions about their knowledge on these topics and decision processes. Responses suggested that such self-assessments provide useful information, e.g., whether respondents are likely to have an acceptable basis for answering, and whether clarifying researcher expectations or offering further motivation is likely to be useful. The third study evaluated the effectiveness of cognitive interviewing for determining whether response categories allow an adequate means of expressing information. Results of a round of cognitive interviews suggested that participants had difficulty providing answers to a particular series of questions. Transcripts of these interviews were coded and analyzed to explore an alternative possibility---that cognitive interviewer behavior actually creates the appearance of problems. Instead, it seems that survey interviewing may suppress the visibility of actual problems. Additional methodological analysis explored how cognitive interviewer behavior varied across questions and across interviewers. This study segues into a subsequent chapter that critically appraises the contribution of cognitive interviewing to questionnaire evaluation methods.