Integrating Artificial Intelligence and Clinical Guidelines to Enhance Heart Failure Recognition Using Electronic Health Records in Preoperative Care

Abstract

Clinical decision support systems (CDSS) have evolved with the development of artificial intelligence (AI), advancing from symbolic AI in the 1970s to recent deep learning models that support clinical decision-making. With increasing accessibility to AI-generated insights, new challenges emerge, such as recognizing inherent biases and ensuring healthcare professionals safely and effectively utilize AI tools. Despite extensive research on CDSS, a comprehensive understanding of how AI insights affect clinical decision-making—particularly compared with traditional clinical guidelines—is still necessary.

This dissertation introduced a web-based educational tool to evaluate the effectiveness of the AI tool that integrates machine learning (ML) and clinical guideline-derived risk factors for identifying preoperative heart failure (HF). The study had three key objectives: (1) develop ML models to identify HF from electronic health records (EHR) data and integrate insights into interventions that assist in HF recognition; (2) compare accuracy, time spent reviewing cases, and confidence in decision-making before and after using these interventions; and (3) explore the intervention’s influences on clinical judgment under uncertainty, particularly in complexity HF screening cases when ML advice is correct or incorrect.

The research involved training logistic regression, random forests, and extreme gradient boosting models using expert-labeled EHR data from surgical patients. The logistic regression model was selected for its performance and interpretability, achieving an area under the curve (AUC) of 0.907, with sensitivity and specificity of 0.830 and 0.829, respectively. Model outputs were converted to human-understandable formats and incorporated into the educational tool. The tool employed two ML-based interventions—direct (MLDR) and indirect recommendation (MLIR)—and an evidence-based (EB) intervention that incorporated guideline-based risk factors.

The study assessed intervention effectiveness through a randomized trial with medical students assigned to EB, MLDR, and MLIR groups. Participants reviewed 10 cases using EHR data alone, followed by 10 additional cases with EHR data and the intervention. Pre- and post-intervention accuracy, time spent reviewing cases, and confidence in clinical judgments were measured. The EB and MLDR groups demonstrated moderate effect sizes of 0.534 and 0.510, respectively, while the MLIR group exhibited a smaller effect size of 0.334. The accuracy improvements were 5.0% (p=0.70) for EB and 4.81% (p=0.045) for MLDR. Additionally, time spent significantly declined across all groups, with declines of 1.15, 0.78, and 0.84 minutes in the MLDR, MLIR, and EB groups, respectively.

The study also examined the impact of case complexity and incorrect ML advice on HF recognition accuracy. In low-complexity cases, MLDR and MLIR groups exhibited significant accuracy improvement of 7.4% (p=0.010) and 11.3% (p<0.001), indicating that mistakes were reduced in identifying key risk factors. However, when provided incorrect ML advice in high-complexity cases, both ML groups experienced significant declines in HF recognition accuracy—16.7% (p=0.026) for MLDR and 19.6% (p=0.036) for MLIR. Despite the decreased accuracy, the MLDR group reported a significant confidence increase in clinical judgments of 42.9% (p<0.001), implying confidence can be bolstered by incorrect ML advice. Conversely, the EB group consistently demonstrated increased accuracy in HF recognition, regardless of case complexity or the correctness of ML predictions.

The results of this dissertation research not only highlight the potential use of ML-based interventions to augment clinical decision-making compared with risk factors derived from clinical guidelines, but they also underscore the importance of healthcare professionals building competencies in AI to complement, rather than complicate, decision-making in clinical practice.