Issues in Survey Data Quality: Four Field Experiments.

Abstract

This study addresses four research design issues affecting survey data quality: (1) mode of data collection, (2) systematic interviewer training, (3) survey questionnaire format alterations, and (4) proxy responses. The data were obtained from 14 Michigan school districts, with half-samples r and omly assigned to seven experimental groups. Survey responses were obtained from 3770 of the 5713 former vocational education students for an overall response rate of 66%. Method of Data Collection. Comparisons of data collection methods, i.e., telephone and mail, in districts r and omly assigned to collection mode, revealed: (1) the telephone mode yielded the highest (69.1%); (2) a combination of mail and phone methods yielded a 67.1% response rate; (3) the mail only method brought a significantly lower response (43.3%). However, item response rates were higher for the mailed questionnaires. The two methods obtained significant differences on some demographic variables, e.g., race and sex. These findings suggest that different modes do capture different subclasses of respondents. Systematic Interviewer Training. Interviewers in eight districts were r and omly assigned to either a training or control group. Training group interviewers received a three hour training session followed by a "h and s on" practice session, and an interviewer's manual. Results showed that systematic training was not effective for increasing response rates. Trained interviewers, however, were more persistent in attempting to contact respondents and more successful in obtaining permission to have an outside agency contact the respondent's employer. Survey Questionnaire Format. Respondents in six districts were r and omly assigned to be contacted using either the st and ard questionnaire or a modified version which incorporated alterations of five questions by word or graphic changes. Item and overall response rates were higher using the modified mailed questionnaire form. Similar changes made to the telephone interview schedule did not result in increased response rates. Proxy Response Bias. Proxy response data from two experimental groups were examined to compare proxy and target responses on attitudinal questions and to assess the accuracy of sibling versus "other" proxy types on information about target students. Results revealed that the proxy ratings on attitudinal items were significantly lower than target student ratings, and that sibling responses did not reflect information more accurately than responses of other proxies.