Psychological effects of proper scoring rules

Abstract

Proper scoring rules (PSRs) have been derived to elicit good probability assessments. Because there are so many different kinds of PSRs, this experiment was designed to determine if any particular characteristics contribute to effectiveness. Subjects observed poker chips in jars and bet on the color of the chip to be sampled. On different trials, lists of bets were generated by different PSRs. The type (log, quadratic, or spherical) of PSR used appeared to have essentially no effect on the probability inferred from the bet selected. However, the inferred probability became less extreme with increased steepness in the functions relating score to assessed probability. Also, various suboptimal strategies seemed to be employed when the rule contained both positive and negative scores, so all possible scores should probably be either positive or negative but not both.