The STONE Curve: A ROC‐Derived Model Performance Assessment Tool
dc.contributor.author | Liemohn, Michael | |
dc.contributor.author | Azari, Abigail | |
dc.contributor.author | Ganushkina, Natalia | |
dc.contributor.author | Rastätter, Lutz | |
dc.date.accessioned | 2022-01-05T14:41:53Z | |
dc.date.available | 2022-01-05T14:41:53Z | |
dc.date.issued | 2020-07-15 | |
dc.identifier.citation | Liemohn, M. W., Azari, A. R., Ganushkina, N. Y., & Rastätter, L. (2020). The STONE curve: A ROC‐derived model performance assessment tool. Earth and Space Science, 7, e2020EA001106. https://doi. org/10.1029/2020EA001106 | en_US |
dc.identifier.uri | https://hdl.handle.net/2027.42/171098 | en |
dc.description | This is a new data-model comparison technique for assessing a model's ability for event detection, sliding both the model and data event identification threshold. The initial applications are from magnetospheric physics. | en_US |
dc.description.abstract | A new model validation and performance assessment tool is introduced, the sliding threshold of observation for numeric evaluation (STONE) curve. It is based on the relative operating characteristic (ROC) curve technique, but instead of sorting all observations in a categorical classification, the STONE tool uses the continuous nature of the observations. Rather than defining events in the observations and then sliding the threshold only in the classifier/model data set, the threshold is changed simultaneously for both the observational and model values, with the same threshold value for both data and model. This is only possible if the observations are continuous and the model output is in the same units and scale as the observations, that is, the model is trying to exactly reproduce the data. The STONE curve has several similarities with the ROC curve—plotting probability of detection against probability of false detection, ranging from the (1,1) corner for low thresholds to the (0,0) corner for high thresholds, and values above the zero‐intercept unity‐slope line indicating better than random predictive ability. The main difference is that the STONE curve can be nonmonotonic, doubling back in both the x and y directions. These ripples reveal asymmetries in the data‐model value pairs. This new technique is applied to modeling output of a common geomagnetic activity index as well as energetic electron fluxes in the Earth's inner magnetosphere. It is not limited to space physics applications but can be used for any scientific or engineering field where numerical models are used to reproduce observations. | en_US |
dc.description.sponsorship | The authors would like to thank the U.S. government for sponsoring this research, in particular research grants from NASA (NNX14AC02G, NNX16AG66G, NNX17AI48G, and NNX17AB87G) and NSF (AGS‐1414517). The part of the research done by M. Liemohn and N. Ganushkina received funding from the European Union Horizon 2020 Research and Innovation programme under Grant Agreements 637302 (PROGRESS) and 870452 (PAGER). A. Azari's contributions are based on work supported by the NSF Graduate Research Fellowship Program (DGE 125626C). The SWMF simulations were conducted on the computing facilities at NASA GSFC, and the run output is freely available on their website (https://ccmc.gsfc.nasa.gov/cgi-bin/SWMFpred.cgi) and the CCMC iSWA interactive tool (https://ccmc.gsfc.nasa. gov/iswa/). The real‐time solar wind data were provided by NOAA SWPC (http://www.swpc.noaa.gov/products/real-time-solar-wind). The authors thank the World Data Center in Kyoto, Japan, for the real‐time Dst values (http://wdc.kugi.kyoto-u.ac.jp/dst_realtime/presentmonth/index.html). The IMPTAM simulations are available through the Finnish Meteorological Institute (http://imptam.fmi.fi/) and at the University of Michigan (http://citrine.engin.umich.edu/imptam/). | en_US |
dc.language.iso | en_US | en_US |
dc.publisher | Wiley | en_US |
dc.rights | Attribution 4.0 International | * |
dc.rights.uri | http://creativecommons.org/licenses/by/4.0/ | * |
dc.subject | space physics | en_US |
dc.subject | data-model comparisons | en_US |
dc.subject | event detection metric | en_US |
dc.title | The STONE Curve: A ROC‐Derived Model Performance Assessment Tool | en_US |
dc.type | Article | en_US |
dc.subject.hlbsecondlevel | Atmospheric, Oceanic and Space Sciences | |
dc.subject.hlbtoplevel | Science | |
dc.subject.hlbtoplevel | Engineering | |
dc.description.peerreviewed | Peer Reviewed | en_US |
dc.contributor.affiliationum | Climate and Space Sciences and Engineering, Department of | en_US |
dc.contributor.affiliationumcampus | Ann Arbor | en_US |
dc.description.bitstreamurl | http://deepblue.lib.umich.edu/bitstream/2027.42/171098/1/Liemohn_ESS_2020_STONEcurve.pdf | |
dc.identifier.doi | https://dx.doi.org/10.7302/3774 | |
dc.identifier.source | Earth and Space Science | en_US |
dc.description.mapping | -1 | en_US |
dc.identifier.orcid | 0000-0002-7039-2631 | en_US |
dc.description.filedescription | Description of Liemohn_ESS_2020_STONEcurve.pdf : Main article | |
dc.description.depositor | SELF | en_US |
dc.identifier.name-orcid | Liemohn, Michael; 0000-0002-7039-2631 | en_US |
dc.working.doi | 10.7302/3774 | en_US |
dc.owningcollname | Climate and Space Sciences and Engineering, Department of |
Files in this item
Remediation of Harmful Language
The University of Michigan Library aims to describe library materials in a way that respects the people and communities who create, use, and are represented in our collections. Report harmful or offensive language in catalog records, finding aids, or elsewhere in our collections anonymously through our metadata feedback form. More information at Remediation of Harmful Language.
Accessibility
If you are unable to use this file in its current format, please select the Contact Us link and we can modify it to make it more accessible to you.