Quantitative synthesis methods: Scientific validity and utility for policy. A case study of carotid endarterectomy.

Abstract

Statistical techniques for aggregating results from many research studies are generally termed "research synthesis" or the quantitative synthesis of research. Three separate but closely-related manuscripts examine a number of methodologic issues surrounding the validity of this general statistical approach, particularly when the base of research studies is characterized by "pre-experimental" data. Moreover, validity issues are assessed in the context of utility of the approach for health policy. These issues are examined through a focused study of a costly and controversial surgical procedure, carotid endarterectomy (CE). A quantitative synthesis (QS) or "meta-analysis" of the CE scientific literature (1960 through 1988) was performed. Coding reliability was assessed using a hierarchical approach developed from a conceptual/quantitative model for meta-analytic data. The CE literature base, while sizable, was uneven in terms of methodologic rigor. Of 214 empirical studies published, only a few reported randomization or controls. The first manuscript compared findings from the QS approach with another developing synthesis method, consensus development. To assess convergent validity across methods, CE outcomes were examined against relative rankings from the RAND consensus process on the appropriateness of clinical indications for CE. Data on CE from a traditional review of published studies were first examined as a validation measure of the QS approach; correlation analysis revealed perfect agreement. However, the RAND results regarding appropriate indications for CE surgery correlated less consistently with rates of adverse events found by the QS. The second manuscript examined results of the QS for one particular CE indication, asymptomatic patients. Using multivariate approaches within the meta-analytic dataset, a number of potentially important results emerged regarding relative risk. There were significant differences in outcomes by provider setting, and there was a strong association between improved 30-day outcomes and use of shunting during surgery. The third and final manuscript examined methodologic "threats" to validity encountered in the QS. Statistical and methodologic approaches were developed both to maximize the use of available information and to qualify the validity of the findings. Specifically, three areas of methodologic focus were identified: missing data, valid use of available data (statistical conclusion validity), and understanding variability across outcomes. Multi-center controlled trial data will provide one measure of predictive validity of the QS overall.