Work Description

Date: December 3, 2018

Dataset Title: Data from Model evaluation guidelines for geomagnetic index predictions

Authors: Michael W. Liemohn, James P. McCollough, Miles A. Engel, Vania K. Jordanova, Steven K. Morley

Contact: Mike Liemohn liemohn@umich.edu

Acknowledgment and Supporting Grants:
This paper is the product of the Geomagnetic Indices Working Group of the International CCMC-LWS Working Meeting on Space Weather Metrics. The authors would like to thank the organizers of the workshop for their time and effort to rally the community into action on devising assessment standards for space weather models. We would also like to thank others that contributed but declined authorship, specifically Lutz Rastätter and Leila Mays at NASA and Joshua Rigler at USGS.

The projects leading to these results have received funding from the European Union Seventh Framework Programme (FP7/ 2007-2013) under grant agreement No 606716 SPACESTORM and from the European Union’s Horizon 2020 research and innovation program under grant agreement No 637302 PROGRESS. Work at the University of Michigan was supported by NASA grants NNX14AF34G, NNX17AI48G, NNX17AB87G, 80NSSC17K0015, and NNX14AC02G, and NSF grant 1663770. The Catholic University of America effort was performed under the CUA-NASA Cooperative Agreement supported by NASA Grant NNG11PL10A 670.135. Funding at the University of Sheffield was provided by STFC UK grant ST/R000697/1. The work done at the University of Alcala was supported by grant from MINECO AYA2016-80881-P. SKM acknowledges support from the US Department of Energy's Laboratory Directed Research and Development program (grant number 20170047DR). The work at GFZ Potsdam was supported by Geo.X, the Research Network for Geosciences in Berlin and Potsdam, under Grant No SO_087_GeoX, and by the European Union’s Horizon 2020 research and innovation program under grant agreement No 776287 SWAMI. Work at Los Alamos was supported through the Laboratory Directed Research and Development program by the US Department of Energy under contract DE-AC52-06NA25396. Work at the Institute of Atmospheric Physics was supported by the H2020 COMPET-2017 TechTIDE Project (776011). Work at IRF-Lund was supported by ESA Contract SSA- SWE-P2-1.5

Key Points:
- We review existing practices for assessing geomagnetic index prediction models and recommend a "standard set" of metrics
- Along with fit performance metrics that use all data-model pairs in their formulas, event detection performance metrics are recommended
- Other aspects of metrics assessment best practices, limitations, uncertainties, and geomagnetic index caveats are also discussed

Research Overview:
One aspect of space weather is a magnetic signature across the surface of the Earth. The creation of this signal involves nonlinear interactions of electromagnetic forces on charged particles and can therefore be difficult to predict. The perturbations that space storms and other activity causes in some observation sets, however, are fairly regular in their pattern. Some of these measurements have been compiled together into a single value, a geomagnetic index. Several such indices exist, providing a global estimate of the activity in different parts of geospace.

Models have been developed to predict the time series of these indices, and various statistical methods are used to assess their performance at reproducing the original index. Existing studies of geomagnetic indices, however, use different approaches to quantify the performance of the model. This document defines a standardized set of statistical analyses as a baseline set of comparison tools that are recommended to assess geomagnetic index prediction models. It also discusses best practices, limitations, uncertainties, and caveats to consider when conducting a model assessment.

Methodology:
Three models were assessed against the baseline metrics set discussed in the accompanying paper.

Instrument and/or Software specifications:
The code in these files is written in IDL, Interactive Data Language. The encapsulated postscript files were then combined into the multi-panel figures for the paper using Adobe Illustrator.

Files contained here:
The files below contain the numerical output values, the geomagnetic index values to which they were compared, the IDL codes and resulting encapsulated postscript files, and the combined multi-panel figure files prepared in Illustrator. There is a main directory with the Illustrator files and then three subdirectories, one for each model.

Files in the main directory:
- WINDMI_timeseries.ai: Illustrator file of the multi-panel compilation of the times series data-model comparison plots from the WINDMI model; this is Figure 1 of the accompanying paper in "Space Weather"

- WINDMI_timeseries.tif: 300-dpi TIF image of the Illustrator file above

- ROC_curves.ai: Illustrator file of the multi-panel compilation of the receiver operating characteristic curves for the models; this is Figure 2 of the accompanying paper in "Space Weather"

- ROC_curves.tif: 300-dpi TIF image of the Illustrator file above

- EventMetrics_vThresh.ai: Illustrator file of the multi-panel compilation of the event performance metrics versus event threshold setting; this is Figure 3 of the accompanying paper in "Space Weather"

- EventMetrics_vThresh.tif: 300-dpi TIF image of the Illustrator file above

In the RAMSCB folder:
threshold_stats_v2.txt: ASCII file of event-detection metrics for the RAMSCB model output (ring current atmosphere interaction model with self consistent B-field), the comparison is for the entire month of January 2005

- RAMSCB_comps_ROC.pro: IDL code for making two figures from the threshold_stats values

- RAMSCB_comps_events_Dst_Events.eps: encapsulated postscript file from the IDL code with even-detection metrics as a function of event threshold setting

- RAMSCB_comps_events_Dst_Events.ai: Illustrator version of the encapsulated postscript file

- RAMSCB_comps_events_Dst_ROC.eps: encapsulated postscript file from the IDL code with the receiver operating characteristic curve

- RAMSCB_comps_events_SYMH_ROC.ai: placeholder file while results were in development

- RAMSCB_comps_events_SYMH_vThresh.ai: placeholder file while results were in development

In the RAMSCB/ramdata subfolder:
- kyotodata_Jan2005.txt: observed SYMH index values for the month of January 2005, from the Kyoto World Data Center in Kyoto, Japan

- log_n000000.log: RAMSCB output, including SYMH results, for the simulation of the January 2005 monthlong interval

In the UPOS folder:
- Re_ [Non-DoD Source] Fwd_ 2018SW002067 Research Article Decision Letter.txt: email from James McCollough at the Air Force Research Laboratory, who provided the UPOS Kp prediction model output for this study

- UPOS_Kp_Event_Performance.txt: event-detection metrics results from the UPOS Kp prediction model as a function of event threshold setting, over the 12 year full-solar-cycle interval from 1 October 2001 - 29 July 2013

- UPOS_Kp_Event_Performance_v2.txt: same data as above but with better formatting

- UPOS_comps_ROC.pro: IDL code to make two figures from the Event_Performance values

- UPOS_comps_events_Kp_events.eps: encapsulated postscript file from the IDL code with even-detection metrics as a function of event threshold setting

- UPOS_comps_events_Kp_Events.ai: Illustrator version of the encapsulated postscript file

- UPOS_comps_events_Kp_ROC.eps: encapsulated postscript file from the IDL code with the receiver operating characteristic curve

- upos_output.txt: the Kp values from the UPOS model and from the GFZ Center for Geomagnetism in Potsdam, Germany, for the 12-year interval of the study

In the WINDMI folder:
- OMNI_HRO_1MIN_16617.txt: 1-minute AL and SYMH index values for the entire year of 2014, from the OMNIWeb data base at NASA Goddard Space Flight Center

- OMNI2_H0_MRG1HR_16617.txt: 1-hour AL and Dst index values for the entire year of 2014, from the OMNIWeb data base

- WINDMI_ACE_####.txt: solar wind input quantities for the 19 WINDMI simulations spanning 2014, conducted at the NASA-GSFC Community Coordinated Modeling Center

- WINDMI_Param_####.txt: some other input parameter settings for the 19 WINDMI simulations

- WINDMI_Newell_####.txt: values from the Newell solar wind-magnetosphere coupling function for the 19 WINDMI runs

- WINDMI_Newell_AL_DST_####.txt: AL and Dst output from the 19 WINDMI simulations

- WINDMI_Newell_AL_DST_2014.txt: a compilation of the other similarly-named files in a single output file for all of 2014

- WINDMI_comps_stats.pro: IDL code to make figures from the AL_DST_2014 values

- WINDMI_comps_ROC.pro: IDL code to make the ROC curve from the AL_DST_2014 values

- WINDMI_comps_stats_AL.dat: fit-performance metrics for the WINDMI model against the observed AL values

- WINDMI_comps_stats_AL.eps: encapsulated postscript file of the WINDMI-AL scatterplot comparison of all data-model pairs in 2014

- WINDMI_comps_stats_AL.jpg: JPEG image of the above EPS file

- WINDMI_comps_stats_SYMH.dat: fit-performance metrics for the WINDMI model against the observed SYMH values

- WINDMI_comps_stats_SYMH.eps: encapsulated postscript file of the WINDMI-SYMH scatterplot comparison of all data-model pairs in 2014

- WINDMI_comps_stats_SYMH.jpg: JPEG image of the above EPS file

- WINDMI_comps_events_AL_ROC.eps: encapsulated postscript file of the WINDMI-AL comparison receiver operating characteristic curve

- WINDMI_comps_events_AL_timeseries.eps: encapsulated postscript file of the WINDMI-AL time series comparison figure

- WINDMI_comps_events_AL_vThresh.eps: encapsulated postscript file of the WINDMI-AL event-detection metrics versus threshold setting figures

- WINDMI_comps_events_AL_vThresh.ai: Illustrator version of the EPS file above

- WINDMI_comps_events_SYMH_ROC.eps: encapsulated postscript file of the WINDMI-SYMH comparison receiver operating characteristic curve

- WINDMI_comps_events_SYMH_timeseries.eps: encapsulated postscript file of the WINDMI-SYMH time series comparison figure

- WINDMI_comps_events_SYMH_vThresh.eps: encapsulated postscript file of the WINDMI-SYMH event-detection metrics versus threshold setting figures

- WINDMI_comps_events_SYMH_vThresh.ai: Illustrator version of the EPS file above

Related publication(s):
Liemohn, M. W., et al. (2018). Model evaluation guidelines for geomagnetic index predictions. Space Weather, 16. https://doi.org/10.1029/2018SW002067

Use and Access:
This data is made available under a Creative Commons Public Domain license (CCO 1.0).

To Cite Data:
Liemohn, M. W., et al. (2018). Model evaluation guidelines for geomagnetic index predictions [data set]. University of Michigan Deep Blue Repository. https://doi.org/10.7302/Z25T3HQC