Work Description

Date: 31 January, 2020

Title: The STONE curve: A ROC-derived model performance assessment tool

Authors: Michael W. Liemohn, Abigail R. Azari, Natalia Yu. Ganushkina, and Lutz Rastätter

Contact: Mike Liemohn liemohn@umich.edu

Acknowldegment and Supporting Grants:
The authors would like to thank the US government for sponsoring this research, in particular research grants from NASA (NNX14AC02G, NNX16AG66G, NNX17AI48G, and NNX17AB87G) and NSF (AGS-1414517). The part of the research done by M. Liemohn and N. Ganushkina received funding from the European Union Horizon 2020 Research and Innovation programme under grant agreement 637302 PROGRESS. A. Azari’s contributions are based on work supported by the NSF Graduate Researcher Fellowship Program (DGE 125626C). The SWMF simulations were conducted on the computing facilities at NASA GSFC and the run output is freely available on their website (https://ccmc.gsfc.nasa.gov/cgi-bin/SWMFpred.cgi) and the CCMC iSWA interactive tool (https://ccmc.gsfc.nasa.gov/iswa/). The real-time solar wind data was provided by NOAA SWPC (http://www.swpc.noaa.gov/products/real-time-solar-wind). The authors thank the World Data Center in Kyoto, Japan for the real-time Dst values (http://wdc.kugi.kyoto-u.ac.jp/dst_realtime/presentmonth/index.html). The IMPTAM simulations are available through the Finnish Meteorological Institute (http://imptam.fmi.fi/) and at the University of Michigan (http://citrine.engin.umich.edu/imptam/).

Key Points:
- A new event-detection-based metric for model performance appraisal is given with sliding thresholds in both observational and model values
- The new metric is like the relative operating characteristic curve but uses continuous observational values, not just categorical status
- The new metric is used on real-time model predictions of a common geomagnetic activity index, demonstrating its features and strengths

Research Overview:
Scientists often try to reproduce observations with a model, helping them explain the observations by adjusting known and controllable features within the model. They then use a large variety of metrics for assessing the ability of a model to reproduce the observations. One such metric is called the relative operating characteristic (ROC) curve, a tool that assesses a model’s ability to predict events within the data. The ROC curve is made by sliding the event-definition threshold in the model output, calculating certain metrics and making a graph of the results. Here, a new model assessment tool is introduced, called the sliding threshold of observation for numeric evaluation (STONE) curve. The STONE curve is created by sliding the event definition threshold not only for the model output but also simultaneously for the data values. This is applicable when the model output is trying to reproduce the exact values of a particular data set. While the ROC curve is still a highly valuable tool for optimizing the prediction of known and pre-classified events, it is argued here that the STONE curve is better for assessing model prediction of a continuous-valued data set.

Methodology:
Two space environment prediction models were assessed against relevant observations using both a relative operating characteristic (ROC) curve as well as a sliding threshold of observations for numeric evaluation (STONE) curve. Similarities and differences between the two metrics are discussed in the paper.

Instrument and/or Software specifications:
The code in these files is written in IDL, Interactive Data Language, version 8.7.2. The encapsulated postscript files were then combined into the multi-panel figures for the paper using Adobe Illustrator.

Files contained here:

- Combined_IMPTAM_MAGED_OMNIDATA.dat: energetic electron fluxes measured by the magnetosphere electron detector (MAGED) on the geosynchronus orbiting environmental satellites (GOES) in geostationary orbit at 6.62 Earth radii geocentric distance over the American sector, specifically from GOES-13, GOES-14, and GOES-15, along with corresponding output from the inner magnetosphere particle transport and acceleration model (IMPTAM) running in real time at the University of Michigan. Dates of the data and model values span September 20, 2013 through March 31, 2015.

- IMPTAM_GOES_ROCnSTONE.pro: Interactive Data Language (IDL) code that calculates the ROC and STONE curve values for the GOES-IMPTAM comparison

- IMPTAM_GOES__MLTmax24_MLTmin00_STONE__STONEmax06.30_STONEmin03.30_ROC__ROCmax06.30_ROCmin03.30.dat: file created by the IDL routine of the GOES-IMPTAM comparison listing the values plotting the ROC and STONE curves. This output was created for the entire magnetic local time range of 00 to 24.

- IMPTAM_GOES__MLTmax09_MLTmin03_STONE__STONEmax06.30_STONEmin03.30_ROC__ROCmax06.30_ROCmin03.30.dat: file created by the IDL routine of the GOES-IMPTAM comparison listing the values plotting the ROC and STONE curves. This output was created for the limited magnetic local time range of 03 to 09.

- DstKyoto.txt: hourly disturbance storm time index (Dst) values from the Kyoto World Data Center.

- yyyymm_SWMF2011_RT_Dst.txt: output from the Space Weather Modeling Framework (SWMF) of a value that should be equivalent to the disturbance storm time (Dst) index. The interval of comparison spans from 19 April 2015 until 17 July 2017, separated into monthly files (replacing the "yyyymm" of the filename)

- SWMF_Dst_hr_ROCnSTONE.pro: IDL code that calculates the ROC and STONE curve values for the Dst-SWMF comparison

- SWMF-2011_Dst-hr_ROCnSTONE_NoBadMarch_2gaps_ROCmax010_ROCmin120.dat: file created by the IDL routine of the Dst-SWMF comparison listing the values plotting the ROC and STONE curves

Related publication:
Liemohn, M. W., Azari, A. R., Ganushkina, N. Yu., & Rastätter, L. (2020). The STONE curve: A ROC-derived model performance assessment tool. Earth and Space Science, 7, e2020EA001106. https://doi.org/10.2019/2020EA001106

Use and Access:
This data set is made available under a Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0).

To Cite Data:
Liemohn, M., Azari, A., Ganushkina, N., Rastätter, L. (2020). Data for the STONE curve: A ROC-derived model performance assessment tool [Data set]. University of Michigan - Deep Blue. https://doi.org/10.7302/mkx6-p686