The STONE Curve: A ROC‐Derived Model Performance Assessment Tool
dc.contributor.author | Liemohn, Michael W. | |
dc.contributor.author | Azari, Abigail R. | |
dc.contributor.author | Ganushkina, Natalia Y. | |
dc.contributor.author | Rastätter, Lutz | |
dc.date.accessioned | 2020-09-02T15:01:49Z | |
dc.date.available | WITHHELD_12_MONTHS | |
dc.date.available | 2020-09-02T15:01:49Z | |
dc.date.issued | 2020-08 | |
dc.identifier.citation | Liemohn, Michael W.; Azari, Abigail R.; Ganushkina, Natalia Y.; Rastätter, Lutz (2020). "The STONE Curve: A ROC‐Derived Model Performance Assessment Tool." Earth and Space Science 7(8): n/a-n/a. | |
dc.identifier.issn | 2333-5084 | |
dc.identifier.issn | 2333-5084 | |
dc.identifier.uri | https://hdl.handle.net/2027.42/156486 | |
dc.description.abstract | A new model validation and performance assessment tool is introduced, the sliding threshold of observation for numeric evaluation (STONE) curve. It is based on the relative operating characteristic (ROC) curve technique, but instead of sorting all observations in a categorical classification, the STONE tool uses the continuous nature of the observations. Rather than defining events in the observations and then sliding the threshold only in the classifier/model data set, the threshold is changed simultaneously for both the observational and model values, with the same threshold value for both data and model. This is only possible if the observations are continuous and the model output is in the same units and scale as the observations, that is, the model is trying to exactly reproduce the data. The STONE curve has several similarities with the ROC curve—plotting probability of detection against probability of false detection, ranging from the (1,1) corner for low thresholds to the (0,0) corner for high thresholds, and values above the zero‐intercept unity‐slope line indicating better than random predictive ability. The main difference is that the STONE curve can be nonmonotonic, doubling back in both the x and y directions. These ripples reveal asymmetries in the data‐model value pairs. This new technique is applied to modeling output of a common geomagnetic activity index as well as energetic electron fluxes in the Earth’s inner magnetosphere. It is not limited to space physics applications but can be used for any scientific or engineering field where numerical models are used to reproduce observations.Plain Language SummaryScientists often try to reproduce observations with a model, helping them explain the observations by adjusting known and controllable features within the model. They then use a large variety of metrics for assessing the ability of a model to reproduce the observations. One such metric is called the relative operating characteristic (ROC) curve, a tool that assesses a model’s ability to predict events within the data. The ROC curve is made by sliding the event‐definition threshold in the model output, calculating certain metrics and making a graph of the results. Here, a new model assessment tool is introduced, called the sliding threshold of observation for numeric evaluation (STONE) curve. The STONE curve is created by sliding the event definition threshold not only for the model output but also simultaneously for the data values. This is applicable when the model output is trying to reproduce the exact values of a particular data set. While the ROC curve is still a highly valuable tool for optimizing the prediction of known and preclassified events, it is argued here that the STONE curve is better for assessing model prediction of a continuous‐valued data set.Key PointsA new event‐detection‐based metric for model performance appraisal is given with sliding thresholds in both observational and model valuesThe new metric is like the relative operating characteristic curve but uses continuous observational values, not just categorical statusThe new metric is used on real‐time model predictions of common geomagnetic activity parameters, demonstrating its features and strengths | |
dc.publisher | Wiley‐Blackwell | |
dc.subject.other | forecasting | |
dc.subject.other | model validation | |
dc.subject.other | data‐model comparison | |
dc.subject.other | STONE curve | |
dc.subject.other | ROC curve | |
dc.title | The STONE Curve: A ROC‐Derived Model Performance Assessment Tool | |
dc.type | Article | |
dc.rights.robots | IndexNoFollow | |
dc.subject.hlbsecondlevel | Atmospheric and Oceanic Sciences | |
dc.subject.hlbsecondlevel | Geological Sciences | |
dc.subject.hlbsecondlevel | Space Sciences | |
dc.subject.hlbtoplevel | Science | |
dc.description.peerreviewed | Peer Reviewed | |
dc.description.bitstreamurl | http://deepblue.lib.umich.edu/bitstream/2027.42/156486/2/ess2610.pdf | en_US |
dc.description.bitstreamurl | http://deepblue.lib.umich.edu/bitstream/2027.42/156486/1/ess2610_am.pdf | en_US |
dc.identifier.doi | 10.1029/2020EA001106 | |
dc.identifier.source | Earth and Space Science | |
dc.identifier.citedreference | Murphy, A. H., & Winkler, R. L. ( 1987 ). A general framework for forecast verification. Monthly Weather Review, 115 ( 7 ), 1330 – 1338. https://doi.org/10.1175/1520-0493(1987)115%3C1330:AGFFFV%3E2.0.CO;2 | |
dc.identifier.citedreference | Liemohn, M. W., Ganushkina, N. Y., De Zeeuw, D. L., Rastaetter, L., Kuznetsova, M., Welling, D. T., Toth, G., Ilie, R., Gombosi, T. I., & van der Holst, B. ( 2018 ). Real‐time SWMF and CCMC: Assessing the Dst output from continuous operational simulations. Space Weather, 16, 1583 – 1603. https://doi.org/10.1029/2018SW001953 | |
dc.identifier.citedreference | Liemohn, M. W., & Jazowski, M. ( 2008 ). Ring current simulations of the 90 intense storms during solar cycle 23. Journal of Geophysical Research, 113, A00A17. https://doi.org/10.1029/2008JA013466 | |
dc.identifier.citedreference | Liemohn, M. W., McCollough, J. P., Jordanova, V. K., Ngwira, C. M., Morley, S. K., Cid, C., Tobiska, W. K., Wintoft, P., Ganushkina, N. Y., Welling, D. T., Bingham, S., Balikhin, M. A., Opgenoorth, H. J., Engel, M. A., Weigel, R. S., Singer, H. J., Buresova, D., Bruinsma, S., Zhelavskaya, I., Shprits, Y. Y., & Vasile, R. ( 2018 ). Model evaluation guidelines for geomagnetic index predictions. Space Weather, 16, 2079 – 2102. https://doi.org/10.1029/2018SW002067 | |
dc.identifier.citedreference | Liemohn, M. W., Xu, S., Dong, C., Bougher, S. W., Johnson, B. C., Ilie, R., & De Zeeuw, D. L. ( 2017 ). Ionospheric control of the dawn‐dusk asymmetry of the Mars magnetotail current sheet. Journal of Geophysical Research: Space Physics, 122, 6397 – 6414. https://doi.org/10.1002/2016JA023707 | |
dc.identifier.citedreference | Ma, Y. J., Nagy, A. F., Russell, C. T., Strangeway, R. J., Wei, H. Y., & Toth, G. ( 2013 ). A global multispecies single‐fluid MHD study of the plasma interaction around Venus. Journal of Geophysical Research: Space Physics, 118, 321 – 330. https://doi.org/10.1029/2012JA018265 | |
dc.identifier.citedreference | Mason, I. B. ( 1982 ). A model for assessment of weather forecasts. Australian Meteorological Magazine, 30, 291 – 303. | |
dc.identifier.citedreference | Manzato, A. ( 2005 ). An odds ratio parameterization for ROC diagram and skill score indices. Weather and Forecasting, 20 ( 6 ), 918 – 930. https://doi.org/10.1175/WAF899.1 | |
dc.identifier.citedreference | Manzato, A. ( 2007 ). A note on the maximum Peirce skill score. Weather and Forecasting, 22 ( 5 ), 1148 – 1154. https://doi.org/10.1175/WAF1041.1 | |
dc.identifier.citedreference | Mathieu, J. A., & Aires, F. ( 2018 ). Using neural network classifier approach for statistically forecasting extreme corn yield losses in Eastern United States. Earth and Space Science, 5, 622 – 639. https://doi.org/10.1029/2017EA000343 | |
dc.identifier.citedreference | Meade, B. J., DeVries, P. M. R., Faller, J., Viegas, F., & Wattenberg, M. ( 2017 ). What is better than Coulomb failure stress? A ranking of scalar static stress triggering mechanisms from 10 5 mainshock‐aftershock pairs. Geophysical Research Letters, 44, 11,409 – 11,416. https://doi.org/10.1002/2017GL075875 | |
dc.identifier.citedreference | Morley, S. K., Brito, T. V., & Welling, D. T. ( 2018 ). Measures of model performance based on the log accuracy ratio. Space Weather, 16, 69 – 88. https://doi.org/10.1002/2017SW001669 | |
dc.identifier.citedreference | Muller, R. H. ( 1944 ). Verification of short‐range weather forecasts (a survey of the literature). Bulletin of the American Meteorological Society, 25 ( 1 ), 18 – 27. https://doi.org/10.1175/1520-0477-25.1.18 | |
dc.identifier.citedreference | Murphy, A. H. ( 1996 ). The Finley affair: A signal event in the history of forecast verification. Weather and Forecasting, 11 ( 1 ), 3 – 20. https://doi.org/10.1175/1520-0434(1996)011%3C0003:TFAASE%3E2.0.CO;2 | |
dc.identifier.citedreference | Pulkkinen, A., Rastätter, L., Kuznetsova, M., Singer, H., Balch, C., Weimer, D., Toth, G., Ridley, A., Gombosi, T., Wiltberger, M., Raeder, J., & Weigel, R. ( 2013 ). Community‐wide validation of geospace model ground magnetic field perturbation predictions to support model transition to operations. Space Weather, 11, 369 – 385. https://doi.org/10.1002/swe.20056 | |
dc.identifier.citedreference | Reuter, B., & Schweizer, J. ( 2018 ). Describing snow instability by failure initiation, crack propagation, and slab tensile support. Geophysical Research Letters, 45, 7019 – 7027. https://doi.org/10.1029/2018GL078069 | |
dc.identifier.citedreference | Rostoker, G. ( 1972 ). Geomagnetic indices. Reviews of Geophysics and Space Physics, 10 ( 4 ), 935 – 950. https://doi.org/10.1029/RG010i004p00935 | |
dc.identifier.citedreference | Rowland, W., & Weigel, R. S. ( 2012 ). Intracalibration of particle detectors on a three‐axis stabilized geostationary platform. Space Weather, 10, S11002. https://doi.org/10.1029/2012SW000816 | |
dc.identifier.citedreference | Sillanpaa, I., Ganushkina, N. Y., Dubyagin, S., & Rodriguez, J. V. ( 2017 ). Electron fluxes at geostationary orbit from GOES MAGED data. Space Weather, 15, 1602 – 1614. https://doi.org/10.1002/2017SW001698 | |
dc.identifier.citedreference | Stefanescu, E. R., Patra, A. K., Bursik, M. I., Madankan, R., Pouget, S., Jones, M., Singla, P., Singh, T., Pitman, E. B., Pavolonis, M., Morton, D., Webley, P., & Dehn, J. ( 2014 ). Temporal, probabilistic mapping of ash clouds using wind field stochastic variability and uncertain eruption source parameters: Example of the 14 April 2010 Eyjafjallajökull eruption. Journal of Advances in Modeling Earth Systems, 6, 1173 – 1184. https://doi.org/10.1002/2014MS000332 | |
dc.identifier.citedreference | Stephenson, D. B., Casati, B., Ferro, C. A. T., & Wilson, C. A. ( 2008 ). The extreme dependency score: A non‐vanishing measure for forecasts of rare events. Meteorological Applications, 15 ( 1 ), 41 – 50. https://doi.org/10.1002/met.53 | |
dc.identifier.citedreference | Swets, J. A. ( 1973 ). The relative operating characteristic in psychology. Science, 182 ( 4116 ), 990 – 1000. https://doi.org/10.1126/science.182.4116.990 | |
dc.identifier.citedreference | Toth, G., van der Holst, B., Sokolov, I. V., De Zeeuw, D. L., Gombosi, T. I., Fang, F., Manchester, W. B., Meng, X., Najib, D., Powell, K. G., Stout, Q. F., Glocer, A., Ma, Y.‐J., & Opher, M. ( 2012 ). Adaptive numerical algorithms in space weather modeling. Journal of Computational Physics, 231 ( 3 ), 870 – 903. https://doi.org/10.1016/j.jcp.2011.02.006 | |
dc.identifier.citedreference | Wilks, D. S. ( 2019 ). Statistical methods in the atmospheric sciences ( 4th ed. ). Oxford: Academic Press. | |
dc.identifier.citedreference | Yu, Y., Ridley, A. J., Welling, D. T., & Tóth, G. ( 2010 ). Including gap region field‐aligned currents and magnetospheric currents in the MHD calculation of ground‐based magnetic field perturbations. Journal of Geophysical Research: Space Physics, 115 ( A8 ). https://doi.org/10.1029/2009ja014869 | |
dc.identifier.citedreference | Anagnostopoulos, G. G., Fatichi, S., & Burlando, P. ( 2015 ). An advanced process‐based distributed model for the investigation of rainfall‐induced landslides: The effect of process representation and boundary conditions. Water Resources Research, 51, 7501 – 7523. https://doi.org/10.1002/2015WR016909 | |
dc.identifier.citedreference | Azari, A. R., Liemohn, M. W., Jia, X., Thomsen, M. F., Mitchell, D. G., Sergis, N., Rymer, A. M., Hospodarsky, G. B., Paranicas, C., & Vandegriff, J. ( 2018 ). Interchange injections at Saturn: Statistical survey of energetic H + sudden flux intensifications. Journal of Geophysical Research: Space Physics, 123, 4692 – 4711. https://doi.org/10.1029/2018JA025391 | |
dc.identifier.citedreference | Bobra, M. G., & Couvidat, S. ( 2015 ). Solar flare prediction using SDO/HMI vector magnetic field data with a machine learning algorithm. The Astrophysical Journal, 798 ( 2 ), 135. https://doi.org/10.1088/0004-637X/798/2/135 | |
dc.identifier.citedreference | Borah, N., Sahai, A. K., Chattopadhyay, R., Joseph, S., Abhilash, S., & Goswami, B. N. ( 2013 ). A self‐organizing map–based ensemble forecast system for extended range prediction of active/break cycles of Indian summer monsoon. Journal of Geophysical Research: Atmospheres, 118, 9022 – 9034. https://doi.org/10.1002/jgrd.50688 | |
dc.identifier.citedreference | Boynton, R. J., Amariutei, O. A., Shprits, Y. Y., & Balikhin, M. A. ( 2019 ). The system science development of local time‐dependent 40‐keV electron flux models for geostationary orbit. Space Weather, 17, 894 – 906. https://doi.org/10.1029/2018SW002128 | |
dc.identifier.citedreference | Brenning, A., Grasser, M., & Friend, D. A. ( 2007 ). Statistical estimation and generalized additive modeling of rock glacier distribution in the San Juan Mountains, Colorado, United States. Journal of Geophysical Research, 112, F02S15. https://doi.org/10.1029/2006JF000528 | |
dc.identifier.citedreference | Carter, J. V., Pan, J., Rai, S. N., & Galandiuk, S. ( 2016 ). ROC‐ing along: Evaluation and interpretation of receiver operating characteristic curves. Surgery, 159 ( 6 ), 1638 – 1645. https://doi.org/10.1016/j.surg.2015.12.029 | |
dc.identifier.citedreference | Chen, Y., Manchester, W. B., Hero, A. O., Toth, G., Dufumier, B., Zhou, T., Wang, X., Zhu, H., Sun, Z., & Gombosi, T. I. ( 2019 ). Identifying solar flare precursors using time series of SDO/HMI images and SHARP parameters. Space Weather, 17, 1404 – 1426. https://doi.org/10.1029/2019SW002214 | |
dc.identifier.citedreference | Delle Monache, L., Hacker, J. P., Zhou, Y., Deng, X., & Stull, R. B. ( 2006 ). Probabilistic aspects of meteorological and ozone regional ensemble forecasts. Journal of Geophysical Research, 111, D24307. https://doi.org/10.1029/2005JD006917 | |
dc.identifier.citedreference | Dong, C., Bougher, S. W., Ma, Y., Toth, G., Nagy, A. F., & Najib, D. ( 2014 ). Solar wind interaction with Mars upper atmosphere: Results from the one‐way coupling between the multifluid MHD model and the MTGCM model. Geophysical Research Letters, 41, 2708 – 2715. https://doi.org/10.1002/2014GL059515 | |
dc.identifier.citedreference | Ekelund, S. ( 2011 ). ROC curves—What are they and how are they used? Point of Care, 11 ( 1 ), 16 – 21. https://doi.org/10.1097/POC.0b013e318246a642 | |
dc.identifier.citedreference | Fawcett, T. ( 2006 ). An introduction to ROC analysis. Pattern Recognition Letters, 27 ( 8 ), 861 – 874. https://doi.org/10.1016/j.patrec.2005.10.010 | |
dc.identifier.citedreference | Gabriel, P., Barker, H. W., O’Brien, D., Ferlay, N., & Stephens, G. L. ( 2009 ). Statistical approaches to error identification for plane‐parallel retrievals of optical and microphysical properties of three‐dimensional clouds: Bayesian inference. Journal of Geophysical Research, 114, D06207. https://doi.org/10.1029/2008JD011005 | |
dc.identifier.citedreference | Ganushkina, N. Y., Amariutei, O. A., Shprits, Y. Y., & Liemohn, M. W. ( 2013 ). Transport of the plasma sheet electrons to the geostationary distances. Journal of Geophysical Research: Space Physics, 118, 82 – 98. https://doi.org/10.1029/2012JA017923 | |
dc.identifier.citedreference | Ganushkina, N. Y., Amariutei, O. A., Welling, D., & Heynderickx, D. ( 2015 ). Nowcast model for low‐energy electrons in the inner magnetosphere. Space Weather, 13, 16 – 34. https://doi.org/10.1002/2014SW001098 | |
dc.identifier.citedreference | Ganushkina, N. Y., Liemohn, M. W., Amariutei, O. A., & Pitchford, D. ( 2014 ). Low‐energy electrons (550 keV) in the inner magnetosphere. Journal of Geophysical Research: Space Physics, 119, 246 – 259. https://doi.org/10.1002/2013JA019304 | |
dc.identifier.citedreference | Ganushkina, N. Y., Liemohn, M. W., & Dubyagin, S. ( 2018 ). Current systems in the Earth’s magnetosphere. Reviews of Geophysics, 26, 309 – 332. https://doi.org/10.1002/2017RG000590 | |
dc.identifier.citedreference | Ganushkina, N. Y., Sillanpaa, I., Welling, D. T., Haiducek, J., Liemohn, M. W., Dubyagin, S., & Rodriguez, J. ( 2019 ). Validation of inner magnetosphere particle transport and acceleration model (IMPTAM) on the long‐term GOES MAGED measurements of keV electron fluxes at geostationary orbit. Space Weather, 17, 687 – 708. https://doi.org/10.1029/2018SW002028 | |
dc.identifier.citedreference | Ganushkina, N. Y., Pulkkinen, T. I., Bashkirov, V. F., Baker, D. N., & Li, X. ( 2001 ). Formation of intense nose structures. Geophysical Research Letters, 28, 491 – 494. https://doi.org/10.1029/2000GL011955 | |
dc.identifier.citedreference | Gonzalez, W. D., Joselyn, J. A., Kamide, Y., Kroehl, H. W., Rostoker, G., Tsurutani, B. T., & Vasyliunas, V. M. ( 1994 ). What is a geomagnetic storm?. Journal of Geophysical Research, 99 ( A4 ), 5771 – 5792. https://doi.org/10.1029/93ja02867 | |
dc.identifier.citedreference | Halford, A., Kellerman, A., Garcia‐Sage, K., Klenzing, J., Carter, B., McGranaghan, R., Guild, T., Cid, C., Henney, C., Ganushkina, N., Burrell, A., Terkildsen, M., Thompson, B. J., Pulkkinen, A., McCollough, J., Murray, S., Leka, K. D., Fung, S., Bingham, S., Walsh, B., Liemohn, M., Bisi, M., Morley, S., & Welling, D. ( 2019 ). Application usability levels: A framework for tracking project product progress. Journal of Space Weather and Space Climate, 9, A34. https://doi.org/10.1051/swsc/2019030 | |
dc.identifier.citedreference | Hogan, R. J., & Mason, I. B. ( 2012 ). Deterministic forecasts of binary events. In I. T. Jolliffe, & D. B. Stephenson (Eds.), Forecast verification ( 31 – 60 ). Hoboken, NJ: Wiley‐Blackwell. https://doi.org/10.1002/9781119960003.ch3 | |
dc.identifier.citedreference | Ippolito, A., Scotto, C., Sabbagh, D., Sgrigna, V., & Maher, P. ( 2016 ). A procedure for the reliability improvement of the oblique ionograms automatic scaling algorithm. Radio Science, 51, 454 – 460. https://doi.org/10.1002/2015RS005919 | |
dc.identifier.citedreference | Jia, X., Hansen, K. C., Gombosi, T. I., Kivelson, M. G., Toth, G., DeZeeuw, D. L., & Ridley, A. J. ( 2012 ). Magnetospheric configuration and dynamics of Saturn’s magnetosphere: A global MHD simulation. Journal of Geophysical Research, 117, A05225. https://doi.org/10.1029/2012JA017575 | |
dc.identifier.citedreference | Jolliffe, I. T., & Stephenson, D. B. ( 2012 ). Forecast verification: A practitioner’s guide in atmospheric science. Hoboken, NJ: Wiley‐Blackwell. | |
dc.identifier.citedreference | Katus, R. M., & Liemohn, M. W. ( 2013 ). Similarities and differences in low‐to‐mid‐latitude geomagnetic indices. Journal of Geophysical Research: Space Physics, 118, 5149 – 5156. https://doi.org/10.1002/jgra.50501 | |
dc.identifier.citedreference | Li, X. ( 2004 ). Variations of 0.7–6.0 MeV electrons at geosynchronous orbit as a function of solar wind. Space Weather, 2, S03006. https://doi.org/10.1029/2003SW000017 | |
dc.owningcollname | Interdisciplinary and Peer-Reviewed |
Files in this item
Remediation of Harmful Language
The University of Michigan Library aims to describe library materials in a way that respects the people and communities who create, use, and are represented in our collections. Report harmful or offensive language in catalog records, finding aids, or elsewhere in our collections anonymously through our metadata feedback form. More information at Remediation of Harmful Language.
Accessibility
If you are unable to use this file in its current format, please select the Contact Us link and we can modify it to make it more accessible to you.