Search Results

Search Constraints

Start Over
Filtering by: Creator Liemohn, Michael W. Remove constraint Creator: Liemohn, Michael W. Keyword data-model comparison Remove constraint Keyword: data-model comparison

Search Results

  • Data for Defining an independent reference model for event detection skill scores

    work
    Creator:
    Liemohn, Michael W., Ganushkina, Natalia Y., Welling, Daniel T., and Azari, Abigail R.
    Description:
    When we assess a model's ability to predict observed events, there are many equations to choose from, commonly called metrics, that quantify particular aspects of that data-model relationship. One set of such relationships are called skill scores, in which the value from a metric is compared against the same metric but from a different model, a reference model. For assessing event detection, there are several well-known skill scores, all of which are based on a particular reference model. It is shown here that this reference model is not ideal for assessing a new model's skill because it is, unfortunately, based in part on the new model's performance against the data. It is shown that these well-known skill scores have an ambiguous connection to the underlying metric score. Holding the metric value of the new model constant, there is a range of possible skill scores, and conversely, a given skill score value could result from a range of original metric values. It is recommended to stop using these famous skill scores and instead adopt one of several presented alternatives, all of which are fully independent of the new model. All of the plots for this study were created in Excel spreadsheets. The resulting plot files were then combined into the multi-panel figures for the paper using Adobe Illustrator. Specifically, the "xlsx" files were created using Excel Version 16.94 for the Mac and the "txt" files are were generated with Save As -> Tab Delimited Text format.
    Keyword:
    Skill scores, Heidke skill score, Peirce skill score, Gilbert skill score, event detection analysis, and data-model comparison
    Citation to related publication:
    Liemohn, Michael W., Ganushkina, Natalia Yu., Welling, Daniel T., & Azari, Abigail R. (2025). Defining an independent reference model for event detection skill scores. Submitted to AGU Advances, 21 February 2025, manuscript # 2025AV001xxx.
    Discipline:
    Science