External correspondence: Decompositions of the mean probability score

Abstract

Two evaluative criteria for probabilistic forecasting performance, consistency with the axioms of probability theory and external correspondence with the events that ultimately occur, are distinguished. The mean probability, or Brier score (), is the scoring rule most commonly used to quantify external correspondence. A review is made of methods for decomposing into components that represent distinct and important aspects of external correspondence. Data from an empirical study of forecasting performance are used to illustrate the interpretation of the components of the most recent decomposition of (J. F. Yates, Forecasting performance: A covariance decomposition of the mean probability score. Paper presented at 22nd Annual Meeting of the Psychonomic Society, Philadelphia, November 1981; also an unpublished manuscript). Substantively, the most important finding of the study was a "collapsing" tendency in forecasting behavior, whereby subjects were inclined to report forecasts of .5 when they felt they knew little about the event in question. This finding is problematic because self-reported knowledge was only minimally related to the actual external correspondence of the subjects' forecasts. A survey of uses of decompositions suggests, among other things, that current research typically emphasizes calibration, perhaps to the neglect of other, more important dimensions of external correspondence.