Identification, Verification, and Validation of Epidemiological Models in Public Health Practice

Abstract

Spatiotemporal disease models became more prominent in the literature in the face of the COVID-19 pandemic. Standing at the forefront of policy guidance and decision support, flaws in these models became apparent not only to scientists, but the general public at large.

While validated against synthetic and real case data, disease models are often not subjected to model verification tests, undermining efforts. Verification processes confirm that the model accurately represents the developer’s conceptual description, while validation only checks that the model is an accurate representation of the problem given its intended uses.

We deconstruct this problem by beginning first with model formulation methods: how can existing data be used to improve future modeling efforts and evaluate whether common modeling tasks such as parameter estimation are well-posed? A well-posed model must meet three criteria: (1) a solution exists, (2) the solution is unique, and (3) the solution depends continuously on the data (i.e. initial conditions, input functions, or boundary conditions).

For this work, to guarantee well-posedness we examined the identifiability (i.e. uniqueness of the parameter solution estimates) of a broad class of differential equation models that include time-varying forcing function inputs. We evaluate under what circumstances such forcing function inputs may improve the identifiability of the model parameters, and show that in general, such forcing function inputs will not worsen the structural (theoretical) identifiability of the system. The theory presented guarantees that unique parameter fits to data are possible, before the data is introduced to the model.

We next we review and evaluate common modeling choices for spatiotemporal disease models, including some deployed during the COVID-19 pandemic. We attempt to verify that the model assumptions lead to consistent dynamics on varying spatial resolutions. The most common modeling choice of a per capita infection rate, while computationally cheap, is found to be un-verifiable, with results that may vary as a function of the spatial resolution chosen or the grid size; and as such an alternative but less commonly used modeling formulation is presented: the Averaged Infection model.

Given that the Averaged Infection model is verifiable, we move forward to validating this model against simulated and real-world data on different types of populations: dense homogeneous populations like those found in urban regions versus sparse heterogeneous populations like those found in rural towns. A threshold on mobility is recommended to guarantee the consistency condition for dense populations, while we show that more heterogeneous populations may require a stricter (higher) threshold condition. We also demonstrate that even for the Averaged Infection model, if the mobility level and grid resolution are not in agreement, systematic biases may occur in the parameter and R0 estimates, potentially leading to confusion or incorrect predictions when these models are used in practice for decision-making.

Together this dissertation builds a foundation of trust for future modeling efforts, to generate verifiable reliable policy guidance and decision support.