Anchoring Vignettes for Health Comparisons: The Validity of a Multidimensional IRT Model Approach and Design Improvements Using Visual Vignettes

Abstract

With the increasing popularity of cross-cultural research, researchers are facing a difficult problem: respondents with different backgrounds often use different standards when answering survey questions with ordinal response scales, resulting in incomparability of responses across groups (or Differential Item Functioning, DIF). Among the many techniques to ameliorate the problem, anchoring vignettes – a set of questions that describe hypothetical individuals’ situations related to the measure of interest – become an increasingly popular tool for correcting for DIF in interpersonal and cross-cultural comparisons. The successful use of anchoring vignettes depends on two measurement assumptions: 1) response consistency (RC), which means that respondents rate vignette persons in the same way as they rate themselves; and vignette equivalence (VE), which means that the situations posed in vignettes are perceived similarly across respondents. Despite its widespread use, there are several practical challenges, and the fulfillments of the VE and RC assumptions are not always assured. This dissertation intends to fill three important gaps in the existing literature related to anchoring vignettes: specifically, I examine i) a potential statistical solution to evaluate and correct for DIF; ii) the utility of using visual versus verbal vignettes with respect to their efficiency and assumption fulfilments; and iii) question order effects as a source of response error.

The first study focuses on using anchoring vignettes to correct for DIF caused by differential response styles – respondents’ systematic tendency to use the response scales in a certain way regardless of the question content. Recently, a multidimensional IRT model using anchoring vignette items is proposed by Bolt et al. (2014), which allows controlling for all types of response styles. This present study uses several objective benchmarks to evaluate the validity of this IRT model approach by comparing the model with vignettes (to adjust response styles) and the model without vignettes. Data from the first wave of the Survey of Health, Ageing and Retirement in Europe (SHARE) is used for analysis. Findings from this study indicate that the use of anchoring vignettes in this multidimensional IRT model does not effectively control for response styles in the SHARE data. This may be due to the violations of vignette measurement assumptions. These findings highlight the importance of constructing good vignettes that can meet the assumptions.

The second study examines the use of visual vignettes with images as an alternative design to current verbal vignettes for four health domains – pain, mobility, sleep and affect. To compare the performances of verbal and visual vignettes, this study conducts a web survey experiment, collecting data from various racial / ethnic groups. Their performances are compared using various approaches, including survey time and tests of assumptions. Despite the violations of VE found in both visual and verbal vignettes, the results show that, compared to verbal vignettes, visual vignettes can greatly reduce survey time without the loss of the DIF-adjusting quality. This study shows the great potential of visual vignettes in improving efficiency in the survey designs.

The third study evaluates the survey context effects (specifically question order of self-assessment question relative to vignettes) on self-assessed health. The results show that the question order effects could differ among racial / ethnic groups, suggesting that researchers and practitioners need to be cautious of the context effects when using the vignette first, self-assessment last order, especially in cross-cultural studies.