Machine Learning for Healthcare: Model Development and Implementation in Longitudinal Settings

Abstract

Despite great promise, developing and implementing machine learning (ML) models for healthcare remains a challenging engineering task. The progression of disease generates complex longitudinal data that can be difficult to harness when developing models. Additionally, the practice of medicine is inherently dynamic, meaning that implemented models must be responsive to changes.

This dissertation aims to address some of these challenges. In the first part, we focus on the issues surrounding the development of models for use in the setting of occupational injuries. This field has typically focused on developing models that predict injured patients’ return to work dates using information collected around the time of their injury. We demonstrate that a reformulated model using longitudinal observations has better predictive performance than a baseline representative of the existing approaches.

Parts two and three focus on the implementation of ML models. In the second part, we investigate the phenomena of prospective performance degradation. Although ML models experience degradation over time, the amount of degradation expected and the mechanisms through which degradation occurs are unclear. We introduce methods to formally quantify this degradation. Additionally, we present techniques to isolate the leading causes of this degradation, splitting temporal shift (changes in patients and practice) from information technology (IT) infrastructure shift (differences in the data pipelines serving retrospective model development and prospective implementation). These techniques and ancillary analyses allow model developers to debug models to improve prospective model performance.

In the third part, we focus on the problem of updating risk stratification models that have been integrated into clinical practice. Model developers may seek to maintain or improve ML model performance over time. Thus, model developers might update models as part of their regular maintenance. We focus on how updated models may change the risk stratification of patients, leading to poor clinician-model team performance. We propose a new rank-based compatibility measure for assessing risk stratification model updates. In addition to describing the behavior of this measure, we also introduce a technique for model developers to generate updated models that balance high rank-based compatibility against discriminative performance. Altogether, this work provides model developers with methods to analyze and develop updates for risk stratification models that support clinical decision making.