Show simple item record

Overestimating Outcome Rates: Statistical Estimation When Reliability Is Suboptimal

dc.contributor.authorHayward, Rodney A.en_US
dc.contributor.authorHeisler, Michele M.en_US
dc.contributor.authorAdams, Johnen_US
dc.contributor.authorDudley, R. Adamsen_US
dc.contributor.authorHofer, Timothy P.en_US
dc.date.accessioned2010-06-01T21:51:33Z
dc.date.available2010-06-01T21:51:33Z
dc.date.issued2007-08en_US
dc.identifier.citationHayward, Rodney A.; Heisler, Michele; Adams, John; Dudley, R. Adams; Hofer, Timothy P. (2007). "Overestimating Outcome Rates: Statistical Estimation When Reliability Is Suboptimal." Health Services Research 42(4): 1718-1738. <http://hdl.handle.net/2027.42/74896>en_US
dc.identifier.issn0017-9124en_US
dc.identifier.issn1475-6773en_US
dc.identifier.urihttps://hdl.handle.net/2027.42/74896
dc.identifier.urihttp://www.ncbi.nlm.nih.gov/sites/entrez?cmd=retrieve&db=pubmed&list_uids=17610445&dopt=citationen_US
dc.description.abstractTo demonstrate how failure to account for measurement error in an outcome (dependent) variable can lead to significant estimation errors and to illustrate ways to recognize and avoid these errors. Data Sources . Medical literature and simulation models. Study Design/Data Collection . Systematic review of the published and unpublished epidemiological literature on the rate of preventable hospital deaths and statistical simulation of potential estimation errors based on data from these studies. Principal Findings . Most estimates of the rate of preventable deaths in U.S. hospitals rely upon classifying cases using one to three physician reviewers (implicit review). Because this method has low to moderate reliability, estimates based on statistical methods that do not account for error in the measurement of a “preventable death” can result in significant overestimation. For example, relying on a majority rule rating with three reviewers per case (reliability ∼0.45 for the average of three reviewers) can result in a 50–100 percent overestimation compared with an estimate based upon a reliably measured outcome (e.g., by using 50 reviewers per case). However, there are statistical methods that account for measurement error that can produce much more accurate estimates of outcome rates without requiring a large number of measurements per case. Conclusion . The statistical principles discussed in this case study are critically important whenever one seeks to estimate the proportion of cases belonging to specific categories (such as estimating how many patients have inadequate blood pressure control or identifying high-cost or low-quality physicians). When the true outcome rate is low (<20 percent), using an outcome measure that has low-to-moderate reliability will generally result in substantially overestimating the proportion of the population having the outcome unless statistical methods that adjust for measurement error are used.en_US
dc.format.extent439906 bytes
dc.format.extent3109 bytes
dc.format.mimetypeapplication/pdf
dc.format.mimetypetext/plain
dc.publisherBlackwell Publishing Incen_US
dc.rights© 2006 Health Research and Educational Trusten_US
dc.subject.otherReliabilityen_US
dc.subject.otherStatistical Estimationen_US
dc.subject.otherMeasurement Erroren_US
dc.subject.otherMedical Errorsen_US
dc.subject.otherPreventable Deathsen_US
dc.subject.otherAdverse Eventsen_US
dc.titleOverestimating Outcome Rates: Statistical Estimation When Reliability Is Suboptimalen_US
dc.typeArticleen_US
dc.subject.hlbsecondlevelPublic Healthen_US
dc.subject.hlbtoplevelHealth Sciencesen_US
dc.description.peerreviewedPeer Revieweden_US
dc.contributor.affiliationumDepartments of Internal Medicine & Health Management & Policy, University of Michigan Schools of Medicine & Public Health, Ann Arbor, MI.en_US
dc.contributor.affiliationotherDepartment of Veterans Affairs, VA Center for Practice Management & Outcomes Research, P.O. Box 130170, Ann Arbor, MI 48113-0170en_US
dc.contributor.affiliationotherDepartment of Veterans Affairs, VA Center for Practice Management & Outcomes Research, VA Ann Arbor Healthcare System, Ann Arbor, MIen_US
dc.contributor.affiliationotherDepartment of Internal Medicine, Pulmonary & Critical Care Medicine, University of California, San Francisco, CAen_US
dc.contributor.affiliationotherThe Rand Health Program, Santa Monica, CA.en_US
dc.identifier.pmid17610445en_US
dc.description.bitstreamurlhttp://deepblue.lib.umich.edu/bitstream/2027.42/74896/1/j.1475-6773.2006.00661.x.pdf
dc.identifier.doi10.1111/j.1475-6773.2006.00661.xen_US
dc.identifier.sourceHealth Services Researchen_US
dc.identifier.citedreferenceBravo, G., and L. Potvin. 1991. “ Estimating the Reliability of Continuous Measures with Cronbach's Alpha or the Intraclass Correlation Coefficient: Toward the Integration of Two Traditions.” Journal of Clinical Epidemiology 44 ( 4–5 ): 381 – 90.en_US
dc.identifier.citedreferenceBrennan, T. A. 2000. “ The Institute of Medicine Report on Medical Errors—Could It Do Harm? ” New England Journal of Medicine 342: 1123 – 5.en_US
dc.identifier.citedreferenceBrennan, T. A., L. L. Leape, N. M. Laird, L. Hebert, A. R. Localio, A. G. Lawthers, J. P. Newhouse, P. C. Weiler, and H. H. Hiatt. 1991. “ Incidence of Adverse Events and Negligence in Hospitalized Patients.” New England Journal of Medicine 324 ( 6 ): 370 – 6.en_US
dc.identifier.citedreferenceBrennan, T. A., A. R. Localio, L. L. Leape, N. M. Laird, L. Peterson, H. H. Hiatt, and B. A. Barnes. 1990. “ Identification of Adverse Events Occurring during Hospitalization. A Cross-Sectional Study of Litigation, Quality Assurance, and Medical Records at Two Teaching Hospitals.” Annals of Internal Medicine 112: 221 – 6.en_US
dc.identifier.citedreferenceCarmines, E. G., and R. A. Zeller. 1979. Reliability and Validity Assessment. Beverly Hills, CA: Sage Publications.en_US
dc.identifier.citedreferenceCaroll, R. J., D. Ruppert, and L. A. Stefanski. 1995. Measurement Error in Nonlinear Models. London: Chapman & Hall.en_US
dc.identifier.citedreferenceClayton, D. G. 1991. Models for the Analysis of Cohort and Case Control Studies with Inaccurately Measured Exposures. Statistical Models for Longitudinal Studies in Health. New York: Oxford University Press.en_US
dc.identifier.citedreferenceConcato, J., and A. R. Feinstein. 1997. “ Monte Carlo Methods in Clinical Research: Applications in Multivariable Analysis.” Journal of Investigative Medicine 45: 394 – 400.en_US
dc.identifier.citedreferenceCoory, M., and R. Gibberd. 1998. “ New Measures for Reporting the Magnitude of Small-Area Variation in Rates.” Statistical Medicine 17: 2625 – 34.en_US
dc.identifier.citedreferenceCronbach, L. J. 1990. Essentials of Psychological Testing. 5th Edition. New York: Harper and Row.en_US
dc.identifier.citedreferenceDiehr, P., K. Cain, F. Connell, and E. Volinn. 1990. “ What Is Too Much Variation? The Null Hypothesis in Small-Area Analysis.” Health Services Research 24: 741 – 71.en_US
dc.identifier.citedreferenceDiehr, P., and D. Grembowski. 1990. “ A Small Area Simulation Approach to Determining Excess Variation in Dental Procedure Rates.” American Journal of Public Health 80: 1343 – 8.en_US
dc.identifier.citedreferenceDubois, R. W., and R. H. Brook. 1987. “ Hospital Inpatient Mortality. Is It a Predictor of Quality? ” New England Journal of Medicine 317 ( 26 ): 1674 – 80.en_US
dc.identifier.citedreferenceDubois, R. W., and R. H. Brook. 1988. “ Preventable Deaths: Who, How Often, and Why? ” Annals of Internal Medicine 109: 582 – 9.en_US
dc.identifier.citedreferenceFeiveson, A. H. 2002. “ Power by Simulation.” Stata Journal 2: 107 – 24.en_US
dc.identifier.citedreferenceFleiss, J. L. 1986. The Design and Analysis of Clinical Experiments. New York: Wiley.en_US
dc.identifier.citedreferenceGatsonis, C. A., A. M. Epstein, J. P. Newhouse, S. L. Normand, and B. J. McNeil. 1995. “ Variations in the Utilization of Coronary Angiography for Elderly Patients with an Acute Myocardial Infarction. An Analysis Using Hierarchical Logistic Regression.” Medical Care 33: 625 – 42.en_US
dc.identifier.citedreferenceGatsonis, C., S. L. Normand, C. Liu, and C. Morris. 1993. “ Geographic Variation of Procedure Utilization. A Hierarchical Model Approach.” Medical Care 31: 54 – 9.en_US
dc.identifier.citedreferenceHayward, R. A., and T. P. Hofer. 2001. “ Estimating Hospital Deaths Due to Medical Errors: Preventability Is in the Eye of the Reviewer.” Journal of the American Medical Association 286: 415 – 20.en_US
dc.identifier.citedreferenceHayward, R. A., W. G. Manning Jr., L.F McMahon, and A. M. Bernard. 1994. “ Do Attending or Resident Physician Practice Styles Account for Variations in Hospital Resource Use? ” Medical Care 32: 788 – 94.en_US
dc.identifier.citedreferenceHofer, T. P., S. M. Asch, R. A. Hayward, L. V. Rubenstein, M. M. Hogan, J. Adams, and E. A. Kerr. 2004. “ Profiling Quality of Care: Is There a Role for Peer Review? ” BMC Health Services Research 4: 9.en_US
dc.identifier.citedreferenceHofer, T. P., S. J. Bernstein, S. DeMonner, and R. A. Hayward. 2000. “ Discussion between Reviewers Does Not Improve Reliability of Peer Review of Hospital Quality.” Medical Care 38: 152 – 61.en_US
dc.identifier.citedreferenceHofer, T. P., and R. A. Hayward. 1996. “ Identifying Poor-Quality Hospitals. Can Hospital Mortality Rates Detect Quality Problems for Medical Diagnoses? ” Medical Care 34: 737 – 53.en_US
dc.identifier.citedreferenceHofer, T. P., and R. A. Hayward. 2002. “ Are Bad Outcomes from Questionable Clinical Decisions Preventable Medical Errors? A Case of Cascade Iatrogenesis.” Annals of Internal Medicine 137: 327 – 33.en_US
dc.identifier.citedreferenceHofer, T., R. Hayward, S. Greenfield, E. Wagner, S. H. Kaplan, and W. Manning. 1999. “ The Unreliability of Individual Physician ‘Report Cards’ for Assessing the Costs and Quality of Care of A Chronic Disease.” Journal of the American Medical Association 281 ( 22 ): 2098 – 105.en_US
dc.identifier.citedreferenceHofer, T. P., E. A. Kerr, and R. A. Hayward. 2000. “ What Is an Error? ” Effective Clinical Practice 3: 261 – 9.en_US
dc.identifier.citedreferenceHofer, T., and J. Weissfeld. 1994. “ Designing A Simpler High Blood Cholesterol Case Detection Strategy: Are the Advantages of the NCEP Protocol Worth the Complexity? ” Medical Decision Making 14: 357 – 68.en_US
dc.identifier.citedreferenceHolt, D., J. W. McDonald, and C. J. Skinner. 1991 The Effect of Measurement Error on Event History Analysis. Measurement Errors in Surveys. New York: Wiley.en_US
dc.identifier.citedreferenceInformation Bias. Available at http://hsrd.durham.med.va.gov/eric/notebook/ERICIssue16.pdf. (accessed December 27, 2005)en_US
dc.identifier.citedreferenceInstitute of Medicine. 1999. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Press.en_US
dc.identifier.citedreferenceKrein, S. L., T. P. Hofer, E. A. Kerr, and R. A. Hayward. 2002. “ Who Should We Profile? Examining Diabetes Care Practice Variation among Primary Care Providers, Provider Teams and Healthcare Facilities.” Health Services Research 37 ( 5 ): 1159 – 80.en_US
dc.identifier.citedreferenceLeape, L. L. 2000. “ Institute of Medicine Medical Error Figures Are Not Exaggerated.” Journal of the American Medical AssociationA 284: 95 – 7.en_US
dc.identifier.citedreferenceLeape, L. L., T. A. Brennan, N. Laird, A. G. Lawthers, A. R. Localio, B. A. Barnes, L. Hebert, J. P. Newhouse, P. C. Weiler, and H. Hiatt. 1991. “ The Nature of Adverse Events in Hospitalized Patients. Results of the Harvard Medical Practice Study II.” New England Journal of Medicine 324: 377 – 84.en_US
dc.identifier.citedreferenceLyon, R. G. “Optical Systems Characterization and Analysis Research Project” [accessed December 27, 2005]. Available at http://satjournal.tcom.ohiou.edu/pdf/lyon.pdfen_US
dc.identifier.citedreferenceMcDonald, C. J., M. Weiner, and S. L. Hui. 2000. “ Deaths Due to Medical Errors Are Exaggerated in Institute of Medicine Report.” Journal of the American Medical Association 284: 93 – 5.en_US
dc.identifier.citedreferenceOppenheimer, L., and U. Kher. 1999. “ The Impact of Measurement Error on Comparison of Two Treatments Using A Responder Analysis.” Statistical Medicine 18: 2177 – 88.en_US
dc.identifier.citedreferenceSchulzer, M., D. R. Anderson, and S. M. Drance. 1991. “ Sensitivity and Specificity of a Diagnostic Test Determined by Repeated Observations in the Absence of an External Standard.” Journal of Clinical Epidemiology 44: 1167 – 79.en_US
dc.identifier.citedreferenceShen, W., and T. A. Louis. 2000. “ Triple-Goal Estimates for Disease Mapping.” Statistical Medicine 19: 2295 – 308.en_US
dc.identifier.citedreferenceSkrondal, A., and S. Rabe-Hesketh. 2004. Generalized Latent Variable Modeling: Multilevel, Longitudinal, and Structural Equation Models. Boca Raton, FL: Chapman & Hall/CRC.en_US
dc.identifier.citedreferenceSox, H. C. Jr, and S. Woloshin. 2000. “ How Many Deaths Are Due to Medical Error? Getting the Number Right.” Effective Clinical Practice 3: 277 – 83.en_US
dc.identifier.citedreferenceThomas, E. J., S. R. Lipsitz, D. M. Studdert, and T. A. Brennan. 2002. “ The Reliability of Medical Record Review for Estimating Adverse Event Rates.” Annals Internal Medicine 136: 812 – 6.en_US
dc.identifier.citedreferenceThomas, E. J., D. M. Studdert, H. R. Burstin, E. J. Orav, T. Zeena, E. J. Williams, K. M. Howard, P. C. Weiler, and T. A. Brennan. 2000. “ Incidence and Types of Adverse Events and Negligent Care in Utah and Colorado.” Medical Care 38: 261 – 71.en_US
dc.identifier.citedreferenceThomas, E. J., D. M. Studdert, W. B. Runciman, R. K. Webb, E. J. Sexton, R. M. Wilson, R. W. Gibberd, B. T. Harrison, and T. A. Brennan. 2000. “ A Comparison of Iatrogenic Injury Studies in Australia and the USA. I: Context, Methods, Casemix, Population, Patient and Hospital Characteristics.” International Journal of Quality Health Care 12: 371 – 8.en_US
dc.owningcollnameInterdisciplinary and Peer-Reviewed


Files in this item

Show simple item record

Remediation of Harmful Language

The University of Michigan Library aims to describe library materials in a way that respects the people and communities who create, use, and are represented in our collections. Report harmful or offensive language in catalog records, finding aids, or elsewhere in our collections anonymously through our metadata feedback form. More information at Remediation of Harmful Language.

Accessibility

If you are unable to use this file in its current format, please select the Contact Us link and we can modify it to make it more accessible to you.