When we assess a model's ability to predict observed events, there are many equations to choose from, commonly called metrics, that quantify particular aspects of that data-model relationship. One set of such relationships are called skill scores, in which the value from a metric is compared against the same metric but from a different model, a reference model. For assessing event detection, there are several well-known skill scores, all of which are based on a particular reference model. It is shown here that this reference model is not ideal for assessing a new model's skill because it is, unfortunately, based in part on the new model's performance against the data. It is shown that these well-known skill scores have an ambiguous connection to the underlying metric score. Holding the metric value of the new model constant, there is a range of possible skill scores, and conversely, a given skill score value could result from a range of original metric values. It is recommended to stop using these famous skill scores and instead adopt one of several presented alternatives, all of which are fully independent of the new model.
All of the plots for this study were created in Excel spreadsheets. The resulting plot files were then combined into the multi-panel figures for the paper using Adobe Illustrator. Specifically, the "xlsx" files were created using Excel Version 16.94 for the Mac and the "txt" files are were generated with Save As -> Tab Delimited Text format.
Liemohn, Michael W., Ganushkina, Natalia Yu., Welling, Daniel T., & Azari, Abigail R. (2025). Defining an independent reference model for event detection skill scores. Submitted to AGU Advances, 21 February 2025, manuscript # 2025AV001xxx.
