Overestimating Outcome Rates: Statistical Estimation When Reliability Is Suboptimal
dc.contributor.author | Hayward, Rodney A. | en_US |
dc.contributor.author | Heisler, Michele M. | en_US |
dc.contributor.author | Adams, John | en_US |
dc.contributor.author | Dudley, R. Adams | en_US |
dc.contributor.author | Hofer, Timothy P. | en_US |
dc.date.accessioned | 2010-06-01T21:51:33Z | |
dc.date.available | 2010-06-01T21:51:33Z | |
dc.date.issued | 2007-08 | en_US |
dc.identifier.citation | Hayward, Rodney A.; Heisler, Michele; Adams, John; Dudley, R. Adams; Hofer, Timothy P. (2007). "Overestimating Outcome Rates: Statistical Estimation When Reliability Is Suboptimal." Health Services Research 42(4): 1718-1738. <http://hdl.handle.net/2027.42/74896> | en_US |
dc.identifier.issn | 0017-9124 | en_US |
dc.identifier.issn | 1475-6773 | en_US |
dc.identifier.uri | https://hdl.handle.net/2027.42/74896 | |
dc.identifier.uri | http://www.ncbi.nlm.nih.gov/sites/entrez?cmd=retrieve&db=pubmed&list_uids=17610445&dopt=citation | en_US |
dc.description.abstract | To demonstrate how failure to account for measurement error in an outcome (dependent) variable can lead to significant estimation errors and to illustrate ways to recognize and avoid these errors. Data Sources . Medical literature and simulation models. Study Design/Data Collection . Systematic review of the published and unpublished epidemiological literature on the rate of preventable hospital deaths and statistical simulation of potential estimation errors based on data from these studies. Principal Findings . Most estimates of the rate of preventable deaths in U.S. hospitals rely upon classifying cases using one to three physician reviewers (implicit review). Because this method has low to moderate reliability, estimates based on statistical methods that do not account for error in the measurement of a “preventable death” can result in significant overestimation. For example, relying on a majority rule rating with three reviewers per case (reliability ∼0.45 for the average of three reviewers) can result in a 50–100 percent overestimation compared with an estimate based upon a reliably measured outcome (e.g., by using 50 reviewers per case). However, there are statistical methods that account for measurement error that can produce much more accurate estimates of outcome rates without requiring a large number of measurements per case. Conclusion . The statistical principles discussed in this case study are critically important whenever one seeks to estimate the proportion of cases belonging to specific categories (such as estimating how many patients have inadequate blood pressure control or identifying high-cost or low-quality physicians). When the true outcome rate is low (<20 percent), using an outcome measure that has low-to-moderate reliability will generally result in substantially overestimating the proportion of the population having the outcome unless statistical methods that adjust for measurement error are used. | en_US |
dc.format.extent | 439906 bytes | |
dc.format.extent | 3109 bytes | |
dc.format.mimetype | application/pdf | |
dc.format.mimetype | text/plain | |
dc.publisher | Blackwell Publishing Inc | en_US |
dc.rights | © 2006 Health Research and Educational Trust | en_US |
dc.subject.other | Reliability | en_US |
dc.subject.other | Statistical Estimation | en_US |
dc.subject.other | Measurement Error | en_US |
dc.subject.other | Medical Errors | en_US |
dc.subject.other | Preventable Deaths | en_US |
dc.subject.other | Adverse Events | en_US |
dc.title | Overestimating Outcome Rates: Statistical Estimation When Reliability Is Suboptimal | en_US |
dc.type | Article | en_US |
dc.subject.hlbsecondlevel | Public Health | en_US |
dc.subject.hlbtoplevel | Health Sciences | en_US |
dc.description.peerreviewed | Peer Reviewed | en_US |
dc.contributor.affiliationum | Departments of Internal Medicine & Health Management & Policy, University of Michigan Schools of Medicine & Public Health, Ann Arbor, MI. | en_US |
dc.contributor.affiliationother | Department of Veterans Affairs, VA Center for Practice Management & Outcomes Research, P.O. Box 130170, Ann Arbor, MI 48113-0170 | en_US |
dc.contributor.affiliationother | Department of Veterans Affairs, VA Center for Practice Management & Outcomes Research, VA Ann Arbor Healthcare System, Ann Arbor, MI | en_US |
dc.contributor.affiliationother | Department of Internal Medicine, Pulmonary & Critical Care Medicine, University of California, San Francisco, CA | en_US |
dc.contributor.affiliationother | The Rand Health Program, Santa Monica, CA. | en_US |
dc.identifier.pmid | 17610445 | en_US |
dc.description.bitstreamurl | http://deepblue.lib.umich.edu/bitstream/2027.42/74896/1/j.1475-6773.2006.00661.x.pdf | |
dc.identifier.doi | 10.1111/j.1475-6773.2006.00661.x | en_US |
dc.identifier.source | Health Services Research | en_US |
dc.identifier.citedreference | Bravo, G., and L. Potvin. 1991. “ Estimating the Reliability of Continuous Measures with Cronbach's Alpha or the Intraclass Correlation Coefficient: Toward the Integration of Two Traditions.” Journal of Clinical Epidemiology 44 ( 4–5 ): 381 – 90. | en_US |
dc.identifier.citedreference | Brennan, T. A. 2000. “ The Institute of Medicine Report on Medical Errors—Could It Do Harm? ” New England Journal of Medicine 342: 1123 – 5. | en_US |
dc.identifier.citedreference | Brennan, T. A., L. L. Leape, N. M. Laird, L. Hebert, A. R. Localio, A. G. Lawthers, J. P. Newhouse, P. C. Weiler, and H. H. Hiatt. 1991. “ Incidence of Adverse Events and Negligence in Hospitalized Patients.” New England Journal of Medicine 324 ( 6 ): 370 – 6. | en_US |
dc.identifier.citedreference | Brennan, T. A., A. R. Localio, L. L. Leape, N. M. Laird, L. Peterson, H. H. Hiatt, and B. A. Barnes. 1990. “ Identification of Adverse Events Occurring during Hospitalization. A Cross-Sectional Study of Litigation, Quality Assurance, and Medical Records at Two Teaching Hospitals.” Annals of Internal Medicine 112: 221 – 6. | en_US |
dc.identifier.citedreference | Carmines, E. G., and R. A. Zeller. 1979. Reliability and Validity Assessment. Beverly Hills, CA: Sage Publications. | en_US |
dc.identifier.citedreference | Caroll, R. J., D. Ruppert, and L. A. Stefanski. 1995. Measurement Error in Nonlinear Models. London: Chapman & Hall. | en_US |
dc.identifier.citedreference | Clayton, D. G. 1991. Models for the Analysis of Cohort and Case Control Studies with Inaccurately Measured Exposures. Statistical Models for Longitudinal Studies in Health. New York: Oxford University Press. | en_US |
dc.identifier.citedreference | Concato, J., and A. R. Feinstein. 1997. “ Monte Carlo Methods in Clinical Research: Applications in Multivariable Analysis.” Journal of Investigative Medicine 45: 394 – 400. | en_US |
dc.identifier.citedreference | Coory, M., and R. Gibberd. 1998. “ New Measures for Reporting the Magnitude of Small-Area Variation in Rates.” Statistical Medicine 17: 2625 – 34. | en_US |
dc.identifier.citedreference | Cronbach, L. J. 1990. Essentials of Psychological Testing. 5th Edition. New York: Harper and Row. | en_US |
dc.identifier.citedreference | Diehr, P., K. Cain, F. Connell, and E. Volinn. 1990. “ What Is Too Much Variation? The Null Hypothesis in Small-Area Analysis.” Health Services Research 24: 741 – 71. | en_US |
dc.identifier.citedreference | Diehr, P., and D. Grembowski. 1990. “ A Small Area Simulation Approach to Determining Excess Variation in Dental Procedure Rates.” American Journal of Public Health 80: 1343 – 8. | en_US |
dc.identifier.citedreference | Dubois, R. W., and R. H. Brook. 1987. “ Hospital Inpatient Mortality. Is It a Predictor of Quality? ” New England Journal of Medicine 317 ( 26 ): 1674 – 80. | en_US |
dc.identifier.citedreference | Dubois, R. W., and R. H. Brook. 1988. “ Preventable Deaths: Who, How Often, and Why? ” Annals of Internal Medicine 109: 582 – 9. | en_US |
dc.identifier.citedreference | Feiveson, A. H. 2002. “ Power by Simulation.” Stata Journal 2: 107 – 24. | en_US |
dc.identifier.citedreference | Fleiss, J. L. 1986. The Design and Analysis of Clinical Experiments. New York: Wiley. | en_US |
dc.identifier.citedreference | Gatsonis, C. A., A. M. Epstein, J. P. Newhouse, S. L. Normand, and B. J. McNeil. 1995. “ Variations in the Utilization of Coronary Angiography for Elderly Patients with an Acute Myocardial Infarction. An Analysis Using Hierarchical Logistic Regression.” Medical Care 33: 625 – 42. | en_US |
dc.identifier.citedreference | Gatsonis, C., S. L. Normand, C. Liu, and C. Morris. 1993. “ Geographic Variation of Procedure Utilization. A Hierarchical Model Approach.” Medical Care 31: 54 – 9. | en_US |
dc.identifier.citedreference | Hayward, R. A., and T. P. Hofer. 2001. “ Estimating Hospital Deaths Due to Medical Errors: Preventability Is in the Eye of the Reviewer.” Journal of the American Medical Association 286: 415 – 20. | en_US |
dc.identifier.citedreference | Hayward, R. A., W. G. Manning Jr., L.F McMahon, and A. M. Bernard. 1994. “ Do Attending or Resident Physician Practice Styles Account for Variations in Hospital Resource Use? ” Medical Care 32: 788 – 94. | en_US |
dc.identifier.citedreference | Hofer, T. P., S. M. Asch, R. A. Hayward, L. V. Rubenstein, M. M. Hogan, J. Adams, and E. A. Kerr. 2004. “ Profiling Quality of Care: Is There a Role for Peer Review? ” BMC Health Services Research 4: 9. | en_US |
dc.identifier.citedreference | Hofer, T. P., S. J. Bernstein, S. DeMonner, and R. A. Hayward. 2000. “ Discussion between Reviewers Does Not Improve Reliability of Peer Review of Hospital Quality.” Medical Care 38: 152 – 61. | en_US |
dc.identifier.citedreference | Hofer, T. P., and R. A. Hayward. 1996. “ Identifying Poor-Quality Hospitals. Can Hospital Mortality Rates Detect Quality Problems for Medical Diagnoses? ” Medical Care 34: 737 – 53. | en_US |
dc.identifier.citedreference | Hofer, T. P., and R. A. Hayward. 2002. “ Are Bad Outcomes from Questionable Clinical Decisions Preventable Medical Errors? A Case of Cascade Iatrogenesis.” Annals of Internal Medicine 137: 327 – 33. | en_US |
dc.identifier.citedreference | Hofer, T., R. Hayward, S. Greenfield, E. Wagner, S. H. Kaplan, and W. Manning. 1999. “ The Unreliability of Individual Physician ‘Report Cards’ for Assessing the Costs and Quality of Care of A Chronic Disease.” Journal of the American Medical Association 281 ( 22 ): 2098 – 105. | en_US |
dc.identifier.citedreference | Hofer, T. P., E. A. Kerr, and R. A. Hayward. 2000. “ What Is an Error? ” Effective Clinical Practice 3: 261 – 9. | en_US |
dc.identifier.citedreference | Hofer, T., and J. Weissfeld. 1994. “ Designing A Simpler High Blood Cholesterol Case Detection Strategy: Are the Advantages of the NCEP Protocol Worth the Complexity? ” Medical Decision Making 14: 357 – 68. | en_US |
dc.identifier.citedreference | Holt, D., J. W. McDonald, and C. J. Skinner. 1991 The Effect of Measurement Error on Event History Analysis. Measurement Errors in Surveys. New York: Wiley. | en_US |
dc.identifier.citedreference | Information Bias. Available at http://hsrd.durham.med.va.gov/eric/notebook/ERICIssue16.pdf. (accessed December 27, 2005) | en_US |
dc.identifier.citedreference | Institute of Medicine. 1999. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Press. | en_US |
dc.identifier.citedreference | Krein, S. L., T. P. Hofer, E. A. Kerr, and R. A. Hayward. 2002. “ Who Should We Profile? Examining Diabetes Care Practice Variation among Primary Care Providers, Provider Teams and Healthcare Facilities.” Health Services Research 37 ( 5 ): 1159 – 80. | en_US |
dc.identifier.citedreference | Leape, L. L. 2000. “ Institute of Medicine Medical Error Figures Are Not Exaggerated.” Journal of the American Medical AssociationA 284: 95 – 7. | en_US |
dc.identifier.citedreference | Leape, L. L., T. A. Brennan, N. Laird, A. G. Lawthers, A. R. Localio, B. A. Barnes, L. Hebert, J. P. Newhouse, P. C. Weiler, and H. Hiatt. 1991. “ The Nature of Adverse Events in Hospitalized Patients. Results of the Harvard Medical Practice Study II.” New England Journal of Medicine 324: 377 – 84. | en_US |
dc.identifier.citedreference | Lyon, R. G. “Optical Systems Characterization and Analysis Research Project” [accessed December 27, 2005]. Available at http://satjournal.tcom.ohiou.edu/pdf/lyon.pdf | en_US |
dc.identifier.citedreference | McDonald, C. J., M. Weiner, and S. L. Hui. 2000. “ Deaths Due to Medical Errors Are Exaggerated in Institute of Medicine Report.” Journal of the American Medical Association 284: 93 – 5. | en_US |
dc.identifier.citedreference | Oppenheimer, L., and U. Kher. 1999. “ The Impact of Measurement Error on Comparison of Two Treatments Using A Responder Analysis.” Statistical Medicine 18: 2177 – 88. | en_US |
dc.identifier.citedreference | Schulzer, M., D. R. Anderson, and S. M. Drance. 1991. “ Sensitivity and Specificity of a Diagnostic Test Determined by Repeated Observations in the Absence of an External Standard.” Journal of Clinical Epidemiology 44: 1167 – 79. | en_US |
dc.identifier.citedreference | Shen, W., and T. A. Louis. 2000. “ Triple-Goal Estimates for Disease Mapping.” Statistical Medicine 19: 2295 – 308. | en_US |
dc.identifier.citedreference | Skrondal, A., and S. Rabe-Hesketh. 2004. Generalized Latent Variable Modeling: Multilevel, Longitudinal, and Structural Equation Models. Boca Raton, FL: Chapman & Hall/CRC. | en_US |
dc.identifier.citedreference | Sox, H. C. Jr, and S. Woloshin. 2000. “ How Many Deaths Are Due to Medical Error? Getting the Number Right.” Effective Clinical Practice 3: 277 – 83. | en_US |
dc.identifier.citedreference | Thomas, E. J., S. R. Lipsitz, D. M. Studdert, and T. A. Brennan. 2002. “ The Reliability of Medical Record Review for Estimating Adverse Event Rates.” Annals Internal Medicine 136: 812 – 6. | en_US |
dc.identifier.citedreference | Thomas, E. J., D. M. Studdert, H. R. Burstin, E. J. Orav, T. Zeena, E. J. Williams, K. M. Howard, P. C. Weiler, and T. A. Brennan. 2000. “ Incidence and Types of Adverse Events and Negligent Care in Utah and Colorado.” Medical Care 38: 261 – 71. | en_US |
dc.identifier.citedreference | Thomas, E. J., D. M. Studdert, W. B. Runciman, R. K. Webb, E. J. Sexton, R. M. Wilson, R. W. Gibberd, B. T. Harrison, and T. A. Brennan. 2000. “ A Comparison of Iatrogenic Injury Studies in Australia and the USA. I: Context, Methods, Casemix, Population, Patient and Hospital Characteristics.” International Journal of Quality Health Care 12: 371 – 8. | en_US |
dc.owningcollname | Interdisciplinary and Peer-Reviewed |
Files in this item
Remediation of Harmful Language
The University of Michigan Library aims to describe library materials in a way that respects the people and communities who create, use, and are represented in our collections. Report harmful or offensive language in catalog records, finding aids, or elsewhere in our collections anonymously through our metadata feedback form. More information at Remediation of Harmful Language.
Accessibility
If you are unable to use this file in its current format, please select the Contact Us link and we can modify it to make it more accessible to you.