Measures of Model Performance Based On the Log Accuracy Ratio
dc.contributor.author | Morley, S. K. | |
dc.contributor.author | Brito, T. V. | |
dc.contributor.author | Welling, D. T. | |
dc.date.accessioned | 2018-03-07T18:24:00Z | |
dc.date.available | 2019-03-01T21:00:17Z | en |
dc.date.issued | 2018-01 | |
dc.identifier.citation | Morley, S. K.; Brito, T. V.; Welling, D. T. (2018). "Measures of Model Performance Based On the Log Accuracy Ratio." Space Weather 16(1): 69-88. | |
dc.identifier.issn | 1542-7390 | |
dc.identifier.issn | 1542-7390 | |
dc.identifier.uri | https://hdl.handle.net/2027.42/142454 | |
dc.description.abstract | Quantitative assessment of modeling and forecasting of continuous quantities uses a variety of approaches. We review existing literature describing metrics for forecast accuracy and bias, concentrating on those based on relative errors and percentage errors. Of these accuracy metrics, the mean absolute percentage error (MAPE) is one of the most common across many fields and has been widely applied in recent space science literature and we highlight the benefits and drawbacks of MAPE and proposed alternatives. We then introduce the log accuracy ratio and derive from it two metrics: the median symmetric accuracy and the symmetric signed percentage bias. Robust methods for estimating the spread of a multiplicative linear model using the log accuracy ratio are also presented. The developed metrics are shown to be easy to interpret, robust, and to mitigate the key drawbacks of their more widely used counterparts based on relative errors and percentage errors. Their use is illustrated with radiation belt electron flux modeling examples.Key PointsThe median symmetric accuracy and symmetric signed percentage bias are introduced to address some drawbacks of current metricsThe spread of a multiplicative linear model can be robustly estimated using the log accuracy ratioThe properties of the median symmetric accuracy and the symmetric signed percentage bias are demonstrated on radiation belt examples | |
dc.publisher | John Wiley | |
dc.subject.other | forecasting | |
dc.subject.other | model validation | |
dc.title | Measures of Model Performance Based On the Log Accuracy Ratio | |
dc.type | Article | en_US |
dc.rights.robots | IndexNoFollow | |
dc.subject.hlbsecondlevel | Electrical Engineering | |
dc.subject.hlbtoplevel | Engineering | |
dc.description.peerreviewed | Peer Reviewed | |
dc.description.bitstreamurl | https://deepblue.lib.umich.edu/bitstream/2027.42/142454/1/swe20550.pdf | |
dc.description.bitstreamurl | https://deepblue.lib.umich.edu/bitstream/2027.42/142454/2/swe20550_am.pdf | |
dc.identifier.doi | 10.1002/2017SW001669 | |
dc.identifier.source | Space Weather | |
dc.identifier.citedreference | Stevens, S. S. ( 1946 ). On the theory of scales of measurement. Science, 103 ( 2684 ), 677 – 680. https://doi.org/10.1126/science.103.2684.677 | |
dc.identifier.citedreference | Schiller, Q., Tu, W., Ali, A. F., Li, X., Godinez, H. C., Turner, D. L., … Henderson, M. G. ( 2017 ). Simultaneous event‐specific estimates of transport, loss, and source rates for relativistic outer radiation belt electrons. Journal of Geophysical Research: Space Physics, 122, 3354 – 3373. https://doi.org/10.1002/2016JA023093 | |
dc.identifier.citedreference | Selesnick, R. S., & Blake, J. B. ( 1997 ). Dynamics of the outer radiation belt. Geophysical Research Letters, 24 ( 11 ), 1347 – 1350. https://doi.org/10.1029/97GL51409 | |
dc.identifier.citedreference | Sheskin, D. J. ( 2007 ). Handbook of parametric and nonparametric statistical procedures (4th ed.). New York: Chapman and Hall/CRC. | |
dc.identifier.citedreference | Subbotin, D. A., & Shprits, Y. Y. ( 2009 ). Three‐dimensional modeling of the radiation belts using the versatile electron radiation belt (verb) code. Space Weather, 7, S10001. https://doi.org/10.1029/2008SW000452 | |
dc.identifier.citedreference | Swanson, D. A., Tayman, J., & Barr, C. F. ( 2000 ). A note on the measurement of accuracy for subnational demographic estimates. Demography, 37 ( 2 ), 193 – 201. | |
dc.identifier.citedreference | Swanson, D. A., Tayman, J., & Bryan, T. M. ( 2011 ). MAPE‐R: A rescaled measure of accuracy for cross‐sectional subnational population forecasts. Journal of Population Research, 28 ( 2 ), 225 – 243. https://doi.org/10. 1007/s12546-011-9054-5 | |
dc.identifier.citedreference | Thornes, J. E., & Stephenson, D. B. ( 2001 ). How to judge the quality and value of weather forecast products. Meteorological Applications, 8 ( 3 ), 307 – 314. https://doi.org/10.1017/S1350482701003061 | |
dc.identifier.citedreference | Tofallis, C. ( 2015 ). A better measure of relative prediction accuracy. Journal of the Operational Research Society, 66 ( 8 ), 1352 – 1362. | |
dc.identifier.citedreference | Tsyganenko, N. A. ( 2013 ). Data‐based modelling of the Earth’s dynamic magnetosphere: A review. Annales Geophysicae, 31 ( 10 ), 1745 – 1772. https://doi.org/10.5194/angeo-31-1745-2013 | |
dc.identifier.citedreference | Tu, W., Cunningham, G. S., Chen, Y., Henderson, M. G., Camporeale, E., & Reeves, G. D. ( 2013 ). Modeling radiation belt electron dynamics during GEM challenge intervals with the DREAM3D diffusion model. Journal of Geophysical Research: Space Physics, 118, 6197 – 6211. https://doi.org/10.1002/jgra.50560 | |
dc.identifier.citedreference | Walther, B. A., & Moore, J. L. ( 2005 ). The concepts of bias, precision and accuracy, and their use in testing the performance of species richness estimators, with a literature review of estimator performance. Ecography, 28 ( 6 ), 815 – 829. https://doi.org/10.1111/j.2005.0906-7590.04112.x | |
dc.identifier.citedreference | Weiss, L. A., Thomsen, M. F., Reeves, G. D., & McComas, D. J. ( 1997 ). An examination of the Tsyganenko (T89a) field model using a database of two‐satellite magnetic conjunctions. Journal of Geophysical Research, 102, 4911 – 4918. https://doi.org/10.1029/96JA02876 | |
dc.identifier.citedreference | Welling, D. T. ( 2010 ). The long‐term effects of space weather on satellite operations. Annales Geophysicae, 28 ( 6 ), 1361 – 1367. https://doi.org/10.5194/angeo-28-1361-2010 | |
dc.identifier.citedreference | Wilks, D. S. ( 2006 ). Statistical methods in the atmospheric sciences (2nd ed.). Oxford: Academic Press. | |
dc.identifier.citedreference | Yu, Y., Koller, J., Jordanova, V. K., Zaharia, S. G., Friedel, R. W., Morley, S. K., … Spence, H. E. ( 2014 ). Application and testing of the L * neural network with the self‐consistent magnetic field model of RAM‐SCB. Journal of Geophysical Research: Space Physics, 119, 1683 – 1692. https://doi.org/10.1002/2013JA019350 | |
dc.identifier.citedreference | Zhelavskaya, I. S., Spasojevic, M., Shprits, Y. Y., & Kurth, W. S. ( 2016 ). Automated determination of electron density from electric field measurements on the Van Allen Probes spacecraft. Journal of Geophysical Research: Space Physics, 121, 4611 – 4625. https://doi.org/10.1002/2015JA022132 | |
dc.identifier.citedreference | Zheng, Y., & Rosenfeld, D. ( 2015 ). Linear relation between convective cloud base height and updrafts and application to satellite retrievals. Geophysical Research Letters, 42, 6485 – 6491. https://doi.org/10.1002/2015GL064809 | |
dc.identifier.citedreference | Athanasiu, M. A., Pavlos, G. P., Sarafopoulos, D. V., & Sarris, E. T. ( 2003 ). Dynamical characteristics of magnetospheric energetic ion time series: Evidence for low dimensional chaos. Annales Geophysicae, 21, 1995 – 2010. https://doi.org/10.5194/angeo-21-1995-2003 | |
dc.identifier.citedreference | Baker, D. N., McPherron, R. L., Cayton, T. E., & Klebesadel, R. W. ( 1990 ). Linear prediction filter analysis of relativistic electron properties at 6.6 R E. Journal of Geophysical Research, 95, 15,133 – 15,140. https://doi.org/10.1029/JA095iA09p15133 | |
dc.identifier.citedreference | Belian, R. D., Gisler, G. R., Cayton, T., & Christensen, R. ( 1992 ). High‐ Z energetic particles at geosynchronous orbit during the Great Solar Proton Event Series of October 1989. Journal of Geophysical Research, 97, 16,897 – 16,906. https://doi.org/10.1029/92JA01139 | |
dc.identifier.citedreference | Blake, J., Carranza, P., Claudepierre, S., Clemmons, J., Crain Jr., W. R., Dotan, Y., … Zakrzewski, M. ( 2013 ). The Magnetic Electron Ion Spectrometer (MagEIS) instruments aboard the Radiation Belt Storm Probes (RBSP) spacecraft. Space Science Reviews, 179 ( 1–4 ), 383 – 421. https://doi.org/10.1007/s11214-013-9991-8 | |
dc.identifier.citedreference | Brito, T. V., & Morley, S. K. ( 2017 ). Improving empirical magnetic field models by fitting to in situ data using an optimized parameter approach. Space Weather, 15, 1628 – 1648. https://doi.org/10.1002/2017SW001702 | |
dc.identifier.citedreference | Chen, Y., Friedel, R. H. W., Reeves, G. D., Cayton, T. E., & Christensen, R. ( 2007 ). Multisatellite determination of the relativistic electron phase space density at geosynchronous orbit: An integrated investigation during geomagnetic storm times. Journal of Geophysical Research, 112, A11214. https://doi.org/10.1029/2007JA012314 | |
dc.identifier.citedreference | Coleman, C. D., & Swanson, D. A. ( 2007 ). On MAPE‐R as a measure of cross‐sectional estimation and forecast accuracy. Journal of Economic and Social Measurement, 32, 219 – 233. | |
dc.identifier.citedreference | Déqué, M. ( 2011 ). Deterministic forecasts of continuous variables. In I. T. Jolliffe, & D. B. Stephenson (Eds.), Forecast verification (pp. 77 – 94 ). Chichester, UK: John Wiley. https://doi.org/10.1002/9781119960003.ch5 | |
dc.identifier.citedreference | Drozdov, A. Y., Shprits, Y. Y., Aseev, N. A., Kellerman, A. C., & Reeves, G. D. ( 2017 ). Dependence of radiation belt simulations to assumed radial diffusion rates tested for two empirical models of radial transport. Space Weather, 15 ( 1 ), 150 – 162. https://doi.org/10.1002/2016SW001426 | |
dc.identifier.citedreference | Flores, B. E. ( 1986 ). A pragmatic view of accuracy measurement in forecasting. Omega, 14 ( 2 ), 93 – 98. https://doi.org/10.1016/0305-0483(86)90013-7 | |
dc.identifier.citedreference | Francq, C., & Menvielle, M. ( 1996 ). A model for the Am ( Km ) planetary geomagnetic activity index and application to prediction. Geophysical Journal International, 125, 729 – 746. https://doi.org/10.1111/j. 1365-246X.1996.tb06020.x | |
dc.identifier.citedreference | Friedel, R. H. W., Reeves, G. D., & Obara, T. ( 2002 ). Relativistic electron dynamics in the inner magnetosphere—A review. Journal of Atmospheric and Solar‐Terrestrial Physics, 64, 265 – 282. https://doi.org/10.1016/S1364-6826(01)00088-8 | |
dc.identifier.citedreference | Ginet, G. P., O’Brien, T. P., Huston, S. L., Johnston, W. R., Guild, T. B., Friedel, R., … Su, Y.‐J. ( 2013 ). AE9, AP9 and SPM: New models for specifying the trapped energetic particle and space plasma environment. Space Science Reviews, 179, 579 – 615. https://doi.org/10.1007/s11214-013-9964-y | |
dc.identifier.citedreference | Glocer, A., Tóth, G., Gombosi, T., & Welling, D. ( 2009 ). Modeling ionospheric outflows and their impact on the magnetosphere, initial results. Journal of Geophysical Research, 114, A05216. https://doi.org/10.1029/2009JA014053 | |
dc.identifier.citedreference | Grillakis, M. G., Koutroulis, A. G., & Tsanis, I. K. ( 2013 ). Multisegment statistical bias correction of daily GCM precipitation output. Journal of Geophysical Research: Atmospheres, 118, 3150 – 3162. https://doi.org/10.1002/jgrd.50323 | |
dc.identifier.citedreference | Hampel, F. R. ( 1974 ). The influence curve and its role in robust estimation. Journal of the American Statistical Association, 69 ( 346 ), 383 – 393. | |
dc.identifier.citedreference | Hwang, J., Kim, K. C., Dokgo, K., Choi, E., & Kim, H. P. ( 2015 ). Heliocentric potential (HCP) prediction model for nowscast of aviation radiation dose. Journal of Astronomy and Space Science, 22 ( 1 ), 39 – 44. https://doi.org/10.5140/JASS.2015.32.1.39 | |
dc.identifier.citedreference | Hyndman, R. J., & Anathasopoulos, G. ( 2014 ). Forecasting: Principles and practice. Melbourne, Australia: OTexts. | |
dc.identifier.citedreference | Hyndman, R. J., & Koehler, A. B. ( 2006 ). Another look at measures of forecast accuracy. International Journal of Forecasting, 22 ( 4 ), 679 – 688. https://doi.org/10.1016/j.ijforecast.2006.03.001 | |
dc.identifier.citedreference | Jolliffe, I. T., & Stephenson, D. B. ( 2011 ). Introduction. In I. T. Jolliffe, & D. B. Stephenson (Eds.), Forecast verification (pp. 1 – 9 ). Chichester, UK: John Wiley. https://doi.org/10.1002/9781119960003.ch1 | |
dc.identifier.citedreference | Kim, K.‐C., Shprits, Y., Subbotin, D., & Ni, B. ( 2012 ). Relativistic radiation belt electron responses to GEM magnetic storms: Comparison of CRRES observations with 3‐D VERB simulations. Journal of Geophysical Research, 117, A08221. https://doi.org/10.1029/2011JA017460 | |
dc.identifier.citedreference | Kitchenham, B. A., Pickard, L. M., MacDonell, S. G., & Shepperd, M. J. ( 2001 ). What accuracy statistics really measure [software estimation]. IEE Proceedings – Software, 148 ( 3 ), 81 – 85. https://doi.org/10.1049/ip-sen: 20010506 | |
dc.identifier.citedreference | Kohzadi, N., Boyd, M. S., Kermanshahi, B., & Kaastra, I. ( 1996 ). A comparison of artificial neural network and time series models for forecasting commodity prices. Neurocomputing, 10 ( 2 ), 169 – 181. https://doi.org/https://doi.org/10.1016/0925-2312(95)00020-8 | |
dc.identifier.citedreference | Li, X., Baker, D. N., Temerin, M., Reeves, G., Friedel, R., & Shen, C. ( 2005 ). Energetic electrons, 50 keV to 6 Mev, at geosynchronous orbit: Their responses to solar wind variations. Space Weather, 3, S04001. https://doi.org/10.1029/2004SW000105 | |
dc.identifier.citedreference | Li, Z., Hudson, M., & Chen, Y. ( 2014 ). Radial diffusion comparing a THEMIS statistical model with geosynchronous measurements as input. Journal of Geophysical Research: Space Physics, 119, 1863 – 1873. https://doi.org/10.1002/2013JA019320 | |
dc.identifier.citedreference | Lundstedt, H., Gleisner, H., & Wintoft, P. ( 2002 ). Operational forecasts of the geomagnetic Dst index. Geophysical Research Letters, 29 ( 24 ), 2181. https://doi.org/10.1029/2002GL016151 | |
dc.identifier.citedreference | Makridakis, S. ( 1993 ). Accuracy measures: Theoretical and practical concerns. International Journal of Forecasting, 9 ( 4 ), 527 – 529. https://doi.org/10.1016/0169-2070(93)90079-3 | |
dc.identifier.citedreference | Mauk, B., Fox, N., Kanekal, S., Kessel, R., Sibeck, D., & Ukhorskiy, A. ( 2013 ). Science objectives and rationale for the Radiation Belt Storm Probes mission. Space Science Reviews, 179 ( 1–4 ), 3 – 27. https://doi.org/10.1007/s11214-012-9908-y | |
dc.identifier.citedreference | Menvielle, M., & Berthelier, A. ( 1991 ). The K‐derived planetary indices: Description and availability. Reviews of Geophysics, 29 ( 3 ), 415 – 432. https://doi.org/10.1029/91RG00994 | |
dc.identifier.citedreference | Morley, S. K. ( 2016 ). Alternatives to accuracy and bias metrics based on percentage errors for radiation belt modeling applications (Tech. Rep. LA‐UR‐16‐24592). Los Alamos, NM: Los Alamos National Laboratory. https://doi.org/10.2172/1260362 | |
dc.identifier.citedreference | Morley, S. K., Sullivan, J. P., Carver, M. R., Kippen, R. M., Friedel, R. H. W., Reeves, G. D., & Henderson, M. G. ( 2017 ). Energetic particle data from the global positioning system constellation. Space Weather, 15, 283 – 289. https://doi.org/10.1002/2017SW001604 | |
dc.identifier.citedreference | Morley, S. K., Sullivan, J. P., Henderson, M. G., Blake, J. B., & Baker, D. N. ( 2016 ). The Global Positioning System constellation as a space weather monitor: Comparison of electron measurements with Van Allen Probes data. Space Weather, 14 ( 2 ), 76 – 92. https://doi.org/10.1002/2015SW001339 | |
dc.identifier.citedreference | Murphy, A. H. ( 1993 ). What is a good forecast? An essay on the nature of goodness in weather forecasting. Weather and Forecasting, 8 ( 2 ), 281 – 293. https://doi.org/10.1175/1520-0434(1993)008<0281:WIAGFA>2.0.CO;2 | |
dc.identifier.citedreference | O’Brien, T. P., & McPherron, R. L. ( 2003 ). An empirical dynamic equation for energetic electrons at geosynchronous orbit. Journal of Geophysical Research, 108 ( A3 ), 1137. https://doi.org/10.1029/2002JA009324 | |
dc.identifier.citedreference | Osthus, D., Caragea, P. C., Higdon, D., Morley, S. K., Reeves, G. D., & Weaver, B. P. ( 2014 ). Dynamic linear models for forecasting of radiation belt electrons and limitations on physical interpretation of predictive models. Space Weather, 12, 426 – 446. https://doi.org/10.1002/2014SW001057 | |
dc.identifier.citedreference | Reeves, G. D. ( 2011 ). DREAM: An integrated space radiation nowcast system for natural and nuclear radiation belts, Proceedings of the Advanced Maui Optical and Space Surveillance Technologies Conference (AMOS) Maui, HI, September 14–17, 2011 (p. E2 ). Retrieved from http://permalink.lanl.gov/object/tr?what=info:lanl-repo/lareport/LA-UR-11-06307 | |
dc.identifier.citedreference | Reeves, G. D., Morley, S. K., Friedel, R. H. W., Henderson, M. G., Cayton, T. E., Cunningham, G., … Thomsen, D. ( 2011 ). On the relationship between relativistic electron flux and solar wind velocity: Paulikas and Blake revisited. Journal of Geophysical Research, 116, A02213. https://doi.org/10.1029/2010JA015735 | |
dc.identifier.citedreference | Reeves, G. D., Chen, Y., Cunningham, G. S., Friedel, R. W. H., Henderson, M. G., Jordanova, V. K., … Zaharia, S. ( 2012 ). Dynamic radiation environment assimilation model: DREAM. Space Weather, 10, S03006. https://doi.org/10.1029/2011SW000729 | |
dc.identifier.citedreference | Reeves, G. D., Spence, H. E., Henderson, M. G., Morley, S. K., Friedel, R. H. W., Funsten, H. O., … Niehof, J. T. ( 2013 ). Electron acceleration in the heart of the Van Allen radiation belts. Science, 341 ( 6149 ), 991 – 994. https://doi.org/10.1126/science.1237743 | |
dc.identifier.citedreference | Reikard, G. ( 2011 ). Forecasting space weather: Can new econometric methods improve accuracy? Advances in Space Research, 47 ( 12 ), 2073 – 2080. https://doi.org/https://doi.org/10.1016/j.asr.2011.03.037 | |
dc.identifier.citedreference | Rodriguez, J. V., Sandberg, I., Mewaldt, R. A., Daglis, I. A., & Jiggens, P. ( 2017 ). Validation of the effect of cross‐calibrated goes solar proton effective energies on derived integral fluxes by comparison with stereo observations,. Space Weather, 15, 290 – 309. https://doi.org/10.1002/2016SW001533 | |
dc.identifier.citedreference | Rousseeuw, P. J., & Croux, C. ( 1993 ). Alternatives to the median absolute deviation. Journal of the American Statistical Association, 88 ( 424 ), 1273 – 1283. https://doi.org/10.1080/01621459.1993.10476408 | |
dc.owningcollname | Interdisciplinary and Peer-Reviewed |
Files in this item
Remediation of Harmful Language
The University of Michigan Library aims to describe library materials in a way that respects the people and communities who create, use, and are represented in our collections. Report harmful or offensive language in catalog records, finding aids, or elsewhere in our collections anonymously through our metadata feedback form. More information at Remediation of Harmful Language.
Accessibility
If you are unable to use this file in its current format, please select the Contact Us link and we can modify it to make it more accessible to you.