Search Constraints
Filtering by:
Keyword
data-model comparison
Remove constraint Keyword: data-model comparison
1 - 2 of 2
Number of results to display per page
View results as:
Search Results
-
- Creator:
- Liemohn, Michael W., Ganushkina, Natalia Y., Welling, Daniel T., and Azari, Abigail R.
- Description:
- When we assess a model's ability to predict observed events, there are many equations to choose from, commonly called metrics, that quantify particular aspects of that data-model relationship. One set of such relationships are called skill scores, in which the value from a metric is compared against the same metric but from a different model, a reference model. For assessing event detection, there are several well-known skill scores, all of which are based on a particular reference model. It is shown here that this reference model is not ideal for assessing a new model's skill because it is, unfortunately, based in part on the new model's performance against the data. It is shown that these well-known skill scores have an ambiguous connection to the underlying metric score. Holding the metric value of the new model constant, there is a range of possible skill scores, and conversely, a given skill score value could result from a range of original metric values. It is recommended to stop using these famous skill scores and instead adopt one of several presented alternatives, all of which are fully independent of the new model. All of the plots for this study were created in Excel spreadsheets. The resulting plot files were then combined into the multi-panel figures for the paper using Adobe Illustrator. Specifically, the "xlsx" files were created using Excel Version 16.94 for the Mac and the "txt" files are were generated with Save As -> Tab Delimited Text format.
- Keyword:
- Skill scores, Heidke skill score, Peirce skill score, Gilbert skill score, event detection analysis, and data-model comparison
- Citation to related publication:
- Liemohn, Michael W., Ganushkina, Natalia Yu., Welling, Daniel T., & Azari, Abigail R. (2025). Defining an independent reference model for event detection skill scores. Submitted to AGU Advances, 21 February 2025, manuscript # 2025AV001xxx.
- Discipline:
- Science
-
- Creator:
- Liemohn, Michael W, Azari, Abigail R, Ganushkina, Natalia Yu, and Rastätter, Lutz
- Description:
- Scientists often try to reproduce observations with a model, helping them explain the observations by adjusting known and controllable features within the model. They then use a large variety of metrics for assessing the ability of a model to reproduce the observations. One such metric is called the relative operating characteristic (ROC) curve, a tool that assesses a model’s ability to predict events within the data. The ROC curve is made by sliding the event-definition threshold in the model output, calculating certain metrics and making a graph of the results. Here, a new model assessment tool is introduced, called the sliding threshold of observation for numeric evaluation (STONE) curve. The STONE curve is created by sliding the event definition threshold not only for the model output but also simultaneously for the data values. This is applicable when the model output is trying to reproduce the exact values of a particular data set. While the ROC curve is still a highly valuable tool for optimizing the prediction of known and pre-classified events, it is argued here that the STONE curve is better for assessing model prediction of a continuous-valued data set. and Data and code were created using IDL, but can also be accessed with the open-source Gnu Data Language (GDL; see https://github.com/gnudatalanguage/gdl)
- Keyword:
- ROC curve, STONE curve, data-model comparison, model validation, forecasting, and statistical methods
- Citation to related publication:
- Liemohn, M. W., Azari, A. R., Ganushkina, N. Yu., & Rastätter, L. (2020). The STONE curve: A ROC-derived model performance assessment tool. Earth and Space Science, 7, e2020EA001106. https://doi.org/10.2019/2020EA001106
- Discipline:
- Science