Development and validation of a quality appraisal tool for validity studies (QAVALS)

Abstract

Background: Appraisal of methodological quality of included studies is an important component of conducting systematic reviews. Although several quality appraisal tools are available for intervention studies, fewer tools are available for non-randomized designs, especially for studies of measurement properties. Objectives: The purpose of this study was to develop a quality appraisal tool specific to validity studies (QAVALS) and to examine its reliability and validity. Methods: Following identification of key concepts, an initial list of 34 possible items was developed. Content experts rated each item as either ‘essential’, ‘useful but not essential’, and ‘not necessary’. The content validity ratio (CVR) and content validity index (CVI) were calculated to establish content validity following two rounds of review. Inter-rater and test–retest reliability were assessed by two external reviewers using weighted kappa coefficients. Results: Items below a CVR of 0.50 were eliminated resulting in the modified version with 27 items. Following the second round, the final tool with 24 items was developed. The content validity index of QAVALS was 0.90. QAVALS demonstrated excellent test–retest reliability (k = 0.80–0.84, 95% CI = 0.76–0.90) and good overall inter-rater reliability (k = 0.70, 95% CI = 0.61–0.79). Limitations: Individual item reliability was low for four items. Further research is warranted to examine reliability using larger number of studies and raters with different experience levels. Conclusion: QAVALS is the first quality appraisal tool specifically designed to address common types of validity. The QAVALS demonstrates strong content validity, good overall inter-rater and excellent test–retest reliability.