Search Results

Search Constraints

Start Over
Filtering by: Language English Remove constraint Language: English Language Interactive Data Language Remove constraint Language: Interactive Data Language

Search Results

  • Data for the STONE curve: A ROC-derived model performance assessment tool

    work
    Creator:
    Liemohn, Michael W, Azari, Abigail R, Ganushkina, Natalia Yu, and Rastätter, Lutz
    Description:
    Scientists often try to reproduce observations with a model, helping them explain the observations by adjusting known and controllable features within the model. They then use a large variety of metrics for assessing the ability of a model to reproduce the observations. One such metric is called the relative operating characteristic (ROC) curve, a tool that assesses a model’s ability to predict events within the data. The ROC curve is made by sliding the event-definition threshold in the model output, calculating certain metrics and making a graph of the results. Here, a new model assessment tool is introduced, called the sliding threshold of observation for numeric evaluation (STONE) curve. The STONE curve is created by sliding the event definition threshold not only for the model output but also simultaneously for the data values. This is applicable when the model output is trying to reproduce the exact values of a particular data set. While the ROC curve is still a highly valuable tool for optimizing the prediction of known and pre-classified events, it is argued here that the STONE curve is better for assessing model prediction of a continuous-valued data set. and Data and code were created using IDL, but can also be accessed with the open-source Gnu Data Language (GDL; see  https://github.com/gnudatalanguage/gdl)
    Keyword:
    ROC curve, STONE curve, data-model comparison, model validation, forecasting, and statistical methods
    Citation to related publication:
    Liemohn, M. W., Azari, A. R., Ganushkina, N. Yu., & Rastätter, L. (2020). The STONE curve: A ROC-derived model performance assessment tool. Earth and Space Science, 7, e2020EA001106.  https://doi.org/10.2019/2020EA001106
    Discipline:
    Science

  • Data for Analysis of features in a sliding threshold of observation for numeric evaluation (STONE) curve

    work
    Creator:
    Liemohn, Michael W, Adam, Joshua G, and Ganushkina, Natalia Y
    Description:
    Many statistical tools have been developed to aid in the assessment of a numerical model’s quality at reproducing observations. Some of these techniques focus on the identification of events within the data set, times when the observed value is beyond some threshold value that defines it as a value of keen interest. An example of this is whether it will rain, in which events are defined as any precipitation above some defined amount. A method called the sliding threshold of observation for numeric evaluation (STONE) curve sweeps the event definition threshold of both the model output and the observations, resulting in the identification of threshold intervals for which the model does well at sorting the observations into events and nonevents. An excellent data-model comparison will have a smooth STONE curve, but the STONE curve can have wiggles and ripples in it. These features reveal clusters when the model systematically overestimates or underestimates the observations. This study establishes the connection between features in the STONE curve and attributes of the data-model relationship. The method is applied to a space weather example.
    Keyword:
    space physics, statistical methods, and STONE curve
    Citation to related publication:
    Liemohn, M. W., Adam, J. G., & Ganushkina, N. Y. (2022). Analysis of features in a sliding threshold of observation for numeric evaluation (STONE) curve. Space Weather, 20, e2022SW003102.  https://doi.org/10.1029/2022SW003102
    Discipline:
    Science