SECURMYr'CLAS-SiF'ICATICN OF TH!,-~2z PAGE ('~1 Di-ILI i Em"lrred) REPORT DOCUM ENTATION PAGE BFOREA COPNSTER FORN BEFORE COM1PLETING FORM I. REPORT NUMBER 2. GOVT ACCESSION NO. 3. RECIPENT'S CATALOG NUMBER' 011313-3-T 4. TITLE (and Subititle) 5. TYPE OF REPORT & PERIOD COVERED BAYESIAN DATA ANALYSIS OF GAMBLING PREFERENCES Technical 6. PERFORMING ORG. REPORT NUMBER None 7. AUTHOR(s) 8. CONTRACT OR GRANT NUMBER (5) N00014-67-A-0181-0049 Dirk Wendt 9. PERFORMING ORGANIZATION NAME AND ADDRESS10. PROGRAM ELEMENT. PROJECT. TASK AREA & WORK UNIT NUMBERS Engineering Psychology Laboratory 197 Institute of Science & Technology NR 197-021 ARPA Order No. 2105 Ann Arbor, Michigan 48105 11. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATE Advanced Research Projects Agency 2 N r 2 November 1973 1400 Wilson Boulevard NUMBER PAGES i3.'NUMBER OF'PAGES Arlington, Virginia 51 14. MONITORI,:; AGENiCY NAME AND ADDRESS 15. SECURITY CLASS.(of this report) (if different fro m Contro' ing Of ice) Engineering Psychology Programs Unclassified Office of Naval Research _ _.15a DEtCLASSIFICATION/ DOWNGP:.'O5NG Department of the Navy 5a SCHEDLE Arlington, Virginia -.., ", 16. DISTRIBUTION STATEMVENT (f'j.:.Ts 7.r::) Approved for public release; distribution unlimited. 17. DCSTRi3UTION 5TAT-_.\1 ((of,t a,.; ',,t tt~:; k A: b';.. 20o, if i.,.7r.;tf'm i t r i}. 18. SUP.;LE~.4-,,ENTA,.RY rNCT: None 19. KEY \ OnD S (:.:L o..;; Ct '.": i ).,k N; "a; l.W.ry Bayes's theorem Learning theories Data analysis Choice-among-gambles I z zu. ^A >5TR CTi (( *;,.;!:* { 'r'ti.<u;.i. i ~f ^ /,;!.L.':/., *-/ er) This paper emphasizes the use of Bayesian data analysis for experiments with choices among gambles. In an introductory example, the method is illustrated by a comparison of two learning theories. Special problems arise with the analysis of data from decision making experiments which assume deterministic choice models which cannot be handled by Bayesian analyses. Several ways around these difficulties are suggested, discussed, and demonstrated on two sets of data from choice-among-gambles experiments. I DD FORI 1473 EDo;,.r.:F I NOV 65 IS OBSOLETE 1 JAN 73 Unclassified C, r lI0 V -- '.- -'.- '.'%.;- -.LJI JA C # i.. Ib. - -- - -. -

SECURITY CLASSIFICATION OF THIS PAGE (1 h-n D.tW a E;:t.trD/) - 1..I - - -. - - —, -,, —,,, — T-I —,- --- I SECURITY CLA' 5 i FICA T!O I OF THIS PAGE {l(When Data ttrFn:t c/

Bayesian Data Analysis ofi Gambling Preferences I nro(i uction la;yesian data analysis has been feasible since 17()3 when Rev. Thomas Bayes formulated his theorem (which is just a straightforward application of the definition of conditional probability): P(HID) = P(D H) P(H) / Z P(DIH.) P(H.) 1 1 i P(D) (overall prob.) Despite its availability for such a long time, research workers have made little use of it. Even most researchers who consider themselves Bayesians have used it only as a normative model for human information processing but not for processing data, although Edwards, Lindman & Savage (1963) have pointed out its advantages for statistical inference almost 10 years ago, and although easily readable textbooks are available now (e.g., Hays & Winkler 1970 have a long chapter on Bayesian inference, and the books by McGee (1971) and Winkler (1972) are especially devoted to these procedures). Bayesian statistics differs from traditional statistics in using information not contained in the sample, namely, P(H), the prior probability of the hypothesis. In testing hypotheses, traditional statisticians use only P(DIH), rejecting.a hypothesis Hi when P(DJHi) plus the probability of more extreme data is below a certain prefixed level c, Traditional statisticians have occasionally objected to the idea of 1

taking into account any prior information, like P(Hi), which was not obtained from an observed sample. Those who use Bayesian methods but insist upon priors inferred from previous observations rather than intuition call themselves Emnpirical Bayesians (e.g., Martiz, 1970)4 In a sense, Bayesian statistics can be viewed as an extension of traditional statistics; it uses the same information plus something more, namely prior probabilities, under assumption that all information available should be used for decisions among competing hypotheses. Actually, according to the principle of stable estimation, even strongly biassed priors cannot do much harm to the posteriors as long as the data used for their revision do have enough diagnostic impact, and as long as the prior distribution is not too small in the region favored by the data, and/or not too peaked elsewhere. (For more details about the principle of stable estimation, see Edwards, Lindman & Savage, 1963.) Thus, the arbitrary and intuitive nature of prior distributions does not constitute a reason for not using Bayesian statistical methods. It is probably easy to show that every scientist observing and analyzing data has some priors with respect to his hypotheses —however, to discuss this is not the point of this paper, and the reader interested in these problems is referred, e.g., to Kuhn (1962). Convenient techniques to elicit and assess the scientist's prior probability distributions over hypotheses are available; some of them are described, e.g., in Winkler (1967) and Stael von Holstein (1970). In this paper, we pay little attention to prior distributions over 2

hypotheses. We will rather concentrate on likelihoods P(DI|H), which are more public and less controversial than prior P(H.). Usually, a hypothesis to be tested in traditional statistics implies that a certain parameter value obtains, e.g., in traditional null hypothesis testing the hypothesis is: H:0 = 0 for some parameter 0, which is tested against O o the rather diffuse alternative that 90 9. In most cases, traditional stato isticians cannot figure a probability for the data observed given this diffuse alternative hypothesis, and therefore P, the probability of an error type II, is left unknown. In such a case, the Bayesian usually would not consider a point hypothesis 0 = 9 as opposed to a continuum of other values of 9, but rather would assess o a continuous prior distribution over the whole parameter space, which is then treated as a continuous set of hypotheses. The evidence from the sample observed would then be used to revise this continuous prior distribution over the parameter space according the Bayes's theorem, which reads for the continuous case: f(9lx) - g(xle) f(e) fg(xle') f(e') dte and gives a continuous posterior distribution over the same parameter space. Although Bayesian statistics can handle any number of competing hypotheses simultaneously-up to an infinite number which is the continuous case discussed just above-the most convenient case deals with only two competing hypothesessuch as the traditional test of H against its alternative, the catch-all hypothesis. The advantage of testing only two hypotheses against each other in

Bayesian analysis is so that P(D) cancels P(H ID) P(H ID) 2 that Bayes's theorem can then be written in ratio form out: P( H) P(DIH1) P(H2) P(DIH2) This is known as the odds-likelihood-ratio form of Bayes's theorem: 2 = 2 * LR(D); in words: D o posterior odds = prior odds x likelihood ratio, For conditionally independent data, the likelihood for the whole set of data D =(dl, d2..., d ) is the product of the likelihoods of the individual data 1 2 m d: J P(D H.) = i P(d Hi), and then the odds-likelihood-ratio equation becomes: Q = Q D o * I LR(d.). J J a Bayesian data analysis with these formulae are easy, straightforward, and efficient if you have perfect knowledge of the data generating process which gives you P(DIH), but can be quite a problem if you don't. Bayesian Analysis of Learning Data Let's look at an easy case first: excellent examples to do Bayesian data analyses are comparisons of learning models. Eog., Restle & Greeno (1970) 4

compare a linear operator model (H ) by Bower (1961) (also, see Atkinson, Bower & Corothers, 1965, p. 91). P (clHl) = a - (a - b) (1 - 0) and an accumulative model (H2) b + 0 a(n - 1) P(cIH) *n 2 1 + 02(n - 1) where P (c[Hi) is the probability of a correct response on trial n under the n i respective models, 0i is a parameter of the learning curve, and a and b are initial and asymptotic success probabilities, respectively. Corresponding probabilities of wrong responses (errors) are P (elHi) = 1 - P (c|H ). n n i Bower (1961) had 29 Ss learn a list of 10 items, "to a criterion of 2 consecutive errorless cycles. A response was obtained from the S on each presentation of an item" (p. 528). Stimuli were pairs of consonant letters, responses were the integers 1 and 2, each of the assigned to 5 of the stimuli. Twenty-nine Ss times 10 items makes 290 on each trial (unless some Ss did not get to the last trials because they completed their two errorless cycles earlier). The data Bower obtained, in terms of relative frequencies of correct responses on the n-th trial, are reproduced in Table 1, column 2, from Restle & Greeno (1970, p. 8). To evaluate the two competing learning theories H1 and H given the evidence from these data, Restle & Greeno (1970) assumed a = 1, and b =.5, estimated 0i from the data, and calculated P (cH 2) using these parameter estimates. Resulting P (c|H1), Pn(cH 2), and corresponding P (elH1) and P (e|H2) are 5

Table 1: Bayesian analysis of Bower's data from Restle & Greeno (197.0) (1) (2) (3) (4) (5) (6) (7) (8)(9) (10) (11) (12) (13) o CMH,,, _J w Q S S "*"^ ~, —! c. —!::: 0 (2) (IH t. 0,-4. -O aI. *C).H.4 a).U).H - bD 0-. bD C Q C- lHC >*Hc,3 -— ) *H0) *Hg-' g P*O * rP-_l *_. HC^ 'J..,: *, H.,: H-r ( I 11 0 II 0 11 0 II 0 11 ' (I 11 0 11 I + +3 H P t T. *):: ) pH t -* H t_"H p,-. H _. —,. —.. 1.50.50.50.50.50 -.5010 -.5010 -.53010 -.5010 145 145 0 2.67.67.74 55 33.26 -.1759 -.1308 -.4815 -.5850 194 96 1.5746 3.80.78.84.22.16 -.1079 -.0757 -.6576 -.7959 232 58 0.5510 4.85.85.87.15.13 -.0706 -.o605 -.8239 -.8861 247 43 0.1799 5. 90.90.89.10.11 -.0458 -.0506 -1.0000 -.9586 261 29 0.0522 6.93.93.91.07.09 -.0515 -.0410 -1.1549 -1.0458 270 20 0.53830 7.95.96.92.04.08 -.0177 -.0362 -1.3979 -1.0969 276 14 0.8920 8.96.97.93.03.07 -.0152 -.0315 -1.5229. -1.1549 278 12 0.6714 9 -97.98.94.02.06 -.0088 -.0269 -1.6990 -1.2218 281 9 0.7915 10.98.99.95.01.05 -.0044 -.0223 -2.0000 -1.53010 284 6 0.8896 11.99.99.95.01.05 -.0044 -.0223 -2.0000 -1. 3010 287 5 5.0405 log LR = 9.0255 0 -t r — I!-. I

reproduced in columns 3-6 of Table 1. Restle & Greeno then compared the two models by calculating the sum 2 Ai = (P (c[Hi) - P (c observed)) n for both models (i = 1, 2). A1 was.0042, A2 was.011, indicating a better fit of H1. A Bayesian data analysis would consist of calculating likelihood ratios P (c|H)/P (c|H2) for each correct response observed, and P (elH1) / P (e|H2) n 1 2 n n 2 for each error response, and multiplying them all together to get the overall likelihood ratio. To do so, we need absolute frequencies of errors and correct responses on the 11 trials, which are not given in Restle & Greeno's book, nor in Bower's paper. We reconstructed them by multiplying the relative frequencies given in Restle & Greeno (column 2 in Table 1) by 290 (29 Ss times 10 items), resulting in the absolute frequencies of correct responses of f (c) and errors (f (e)) n n reproduced in columns 11 and 12 of Table 1. (These estimates may contain some errors if some Ss quit before reaching the 11th trial because they had completed their two errorless cycles earlier.) For convenience, the calculation of LR(d.) and LR(D) is performed in logJ arithms: In column 13, we have log LR(D ) = f (c) [log P (cIH ) - log P (c H2)] ) [ n n (eH) + f n(e) [log Pn (eIH1) -log Pn (ejH2)] 7

and Z log LR(D ) = log LR(D), n n with the respective logarithms in columns 7 through 10, and observed frequencies f (c) and f (e) in columns 11 and 12. n n The resulting log LR(D) is 9.0253, indicating a likelihood ratio LR(D) 9 over a billion: LR(D) 1.061 * 109. I.e., if we had assumed equal priors, P(H ) P(H(H) =.5, this would mean that H is over a billion times more likely that H. Although this could be taken as strong evidence for the principle of stable estimation-even very heavily biassed priors would have been corrected,t by such a large likelihood ratio, we have to consider it with some reservation. As we pointed out already, it is doubtful if we can actually assume 290 observations in the last trials (7-11) because some Ss may have quit earlier. Reduction of the numbers of observations in the last trials would reduce LR(D) considerably because trials n = 7 through n = 11 contribute most to LR(D), except for n = 2. Unfortunately, the original complete data are no longer available. However, a letter from Bower assures that these figures actually can be taken as numbers of correct responses assuming that the subjects would not make any more errors had they continued after their last two errorless cycles. Another question is whether we really can assume independence of observations enabling us to multiply likelihoods. Although the observation themselves are clearly obtained independently, the independence assumption for the conditional probabilities P (d IH ) might not hold. n i - 8 -

A way out of this might be not to calculate the whole learning curve for each model, but rather just to predict P +l(d |Hi) from the P (observed so n+l j i n far) by P L(ci Pn l) = (1- O ) P + a, and R + a 0 (R + W) n 2 1 1 n+l n' 2 r (R + a 0( (R + W )) + ( l-a) 02 (R + W)) n 2 1 1 n 2 1 1 In Model 2, this requires an additional assumption about R1 and W1; we used R = W =5 for the calculation of P (c[P,H). Actually, the choice 1 1 n n-1'2 of W = R does not make much of a difference. We use this example to demonstrate a slightly different way of performing the data analysis: In Table 1 we took logarithms of P (ciP,H.) and n n-l i P (elP,Hi) for i = 1, 2, and then subtracted the logarithms of these proban Pn-1 i '' bilities for i = 2 from those for i = 1 (multiplied by the respective numbers of observations); in Table 2 we calculate the likelihood ratios for correct responses and errors directly (by dividing the hit probabilities in column 5, and by dividing the error probabilities in column 6 by those in column 7 to yield column 8), and then take the logarithms of these likelihood ratios for hits and errors (columns 10 and 12) to multiply them to the respective numbers of observations (columns 9 and 11), and sum over these products. The log likelihood ratio is now "only" 2.2508, indicating a likelihood ratio of: 178.2 in favor of Model 1. Of course, taking into account the observed number of correct responses on the previous trial in each calculation of 9

Table 2 (1) (2) ) (4) (5) (6) (7) (8) (9) (10) (11) (12) Pn(hit| Pn(hit|) [:1-(33)]j [1-r(4] [ 7 trial Pn(hit) Pn-l obs.) Pn-l obs.) lPn(m) P) Pnmiss) [=290*(2)] [=log(5)] [=290-(9)] [-log(8)] #n observed predicted by predicted by (hit) predicted by predicted by ( ) f(hit) log LR(hit) f(miss) log LR(miss) Model 1 Model 2* Model 1 Model 2ss) 1.50 145 145 2.67.67.74.9054 ~33..26 1.2692 194 -.0431 96 0.1035 3.80.78.78 1.0000.22.22 1.0000 232 0 58 0 4.85.87.85 1.0235.13.15.8667 247 0.0105 43 -.0621 5.90.90.88 1.0227.10.12 -8533 261 0.0098 29 -.0793 6.93 ~95.92 1.0109.07.08.8750 270 0.0048 20 -.0580 7.95.95.94 1.0106.05.06.8333 276 0.0045 14 -.0793 6.96.97.96 1.0104.03.04.7500 278 0.0043 12 -.1249 9.97.97.96 1.0104.03.04.7500 281 0.0043 9 -.1249 10.98..98.97 1.0103.02.03.6667 284 0.0043 6 -.1761 11.99.99.98 1.0102.01.02.5000 287 0.0045 3 -.5010 -------------------------------------------— 10 --- —----— 2 —.2508______ log___________LR ___ H 0 *assuming R, + W, i 10 E = 2.2508 log LR f;a 178.2

P (c Hi.,P ) brings these probabilities under both models closer to Ihe actual data, and thus levels out differences between them. The resulting likelihood ratio is still large enough to correct even strongly biassed prior odds against Model 1, and now it takes conditioned non-independence into account. The analysis could be further improved by many maximum likelihood extimates for 0 rather than the least squares estimates we took from Restle & Greeno (1970) for this demonstration. However, since the evaluation of learning models is not our main concern in this paper, we will now turn to analyses of choice-among-gambles data. Bayesian Analysis of Gambling Preferences As we have seen, Bayesian data analyses are quite straightforward models that provide us explicit probabilities of occurrence between 0 and 1 for each event we might observe. We have taken learning curves as an example; other feasible examples could be taken from psychophysics, signal detection theory, Lucean & Thurstonean choice theories, etc. However, in analyzing gambling preference data we encounter different problems, particularly with deterministic choice models. Since they require deterministic choices, i.eo, with probabilities 0 and 1, no Bayesian data analysis is feasible under these assumptions. This may be one of the reasons why decision analysts and other scientists strongly advocating Bayesian procedures as normative models for human information processing rather seldom use Bayesian methods in their data analyses: they mostly favor deterministic choice models which prevent them from applying their own principles. 11

We are going to illustrate Bayesian data analyses of choice-among-gambles data on two sets of data here, both borrowed from colleagues: one is from an experiment by Hommers (1973) with normal and educable retarded children of 8, 10, 12, and 14 years of age where it seems rather appropriate to replace the deterministic normative model by a probabilistic one, the other set of data is from an experiment by Seghers, Fryback & Goodman (1973) with adult subjects where the conventional (Lucean) probabilistic choice models might indicate too weak preferences as compared to the choice probabilities inferred from the data. Hommers' Data Hommers (1973) in his dissertation compares choices among bets made by 8, 10, and 12 years old normal children, and 8, 10, 12 and 14 years old educable retarded children. Each set of gambles presented as choice alternatives to the S consisted of 3 bets labelled W, L, and S, respectively, where W indicates the choice with the largest amount to be won but with the smallest winning probability, S the one with the largest winning probability but the smallest amount, and L had medium probability and payoff. Table 3 shows winning probabilities (P), payoffs (V), and expected values (EV) for the three choice alternatives W, L, and S of each of Hommers' 15 stimuli. Stimuli were presented to Ss in form of index cards showing sets of "winning" and "not winning" balls in urns, and displaying the amounts to be won in coins. Subjects made their choice by indicating their favored gamble, which was played thereafter. About 12

half of the Ss in each age and school level had previous experience with choicer: on stimulus cardin with two choice alternatives, so that there are three independent variables::'chocl level (nominal vs. edlucable r-eta.trded), age level, and prior gambling (e:perience vs. no prior- gambling experion(cp-, liommers' data, i.e., frequencies of choices of the alternatives W, L, and S of the 15 stimuli in the 14 groups, are displayed in Table 4. Hommers' analysis of these data consisted of chi square comparisons between these figures, testing various hypotheses about differences in the development of risk vs. safety orientation and EV maximization between the age groups tested and between the normal and educable retarted children. However, since it is assumed that these children follow some probabilistic choice model, it is feasible to apply a BTL choice model to these data, and do a likelihood ratio analysis. Three probabilistic choice models derived from Hommers' hypotheses seem to be naturally applicable in this situation: Ss are either (1) safety oriented, i.e., focussing on the probability of winning, and thus should choose the alternatives with probabilities proportional to their respective winning probabilities, or (2) they are value oriented, and choose with probabilities proportional to the payoffs, or (3) they are expected-value oriented, and choose with probabilities proportional to the expected values of the alternatives. All wins and expected values are positive. Choice probabilities for the alternatives W, L, and S of each stimulus are calculated under the assumption of each of these three models, and displayed in Table 5. In these computations, use has been made of the "auxiliary sums" in the last three columns of Table 3; e.g., in stimulus 1, the sum of the EV 13

Table 3: Hommers' (1975) stimuli: three-alternative choices among bets stmalternative alternative alternative O auxiliary sums stimulus # P W EV P W EV P W EV P W EV 1.1 15 1.5.5 10 5.0.9 5 4.5 1.5 30 11.0 2 53 35 10.5.5 15 7.5.7 10 7.0 1.5 60 25.0 3.1 25 2.5.5 15 4.5.5 10 2.5.9 50 9.5 4.1 15 1.5.7 10 7.0.9 5 4.5 1.7 30 13.0 5.1 35 3.5 3. 25 7.5.5 15 7.5.9 75 18.5 6.1 55 355.5 10 350.7 5 5.5 1.1 50 10.0 7.3 15 45.5 10 5.0.7 5 -55 1.5 30 13.0 8.5 35 10.5.5 20 10.0.7 15 10.5 1.5 70 31.0 9.5 35 17i5.7 25 17.5.9 15 13.5 2.1 75 48.5 10 3 25 7-5.5 15 7.5.9 10 9 1.7 50 24.0 11.3 55 10.5.5 25 12.5.7 15 10.5 1.5 75 33.5 12.3 30 9.0.7 20 14.0.9 10 9.0 1.9 60 32.0 13 53 15 4.5.5 10 5.0.9 5 4.5 1.7 30 14.0 14.5 25 12.5.7 15 10.5.9 5 4.5 2.1 45 27.5 15.1 15 1.5.3 10 3.0.9 5 4.5 1.3 30 9.0 Note: maximal EV underlined; by dashed line where 2 maxima mm_ m _mm _ _ _ _ m H-"

Table 4: Hommer's data: absolute choice frequencies in groups without prior gambling experience V 8 V 10 V 12 S 8 S 10 S 12 S 14 stimulus stiulu n = 1 n = 15 n = 10 n = 8 n = 16 n = 13 n = 18 W L S 'S S S S L S W ' S 1 6 1 8 8 1 4 5 3 0 5 7 0 9 5 4 4 2 5 11 2 8 0 7 4 4 7 3 2 5 4 1 3 1 1 5 8 0 5 6 1 11 3 5 2 8 2 0 1 0 4 6 4 0 4 5 1 10 6 0 7 5 3 10 4 3 4 8 1 10 4 0 7 3 4 1 4 5 7 0 5 8 5 8 7 5 4 2 9 1 0 14 1 6 3 0 5 5 0 11 2 2 9 4 1 11 6 4 2 9 2 4 9 53 4 3 0 5 5 3 8 2 2 9 2 4 12 7 5 3 7 1 9 5 1 5 4 5, 2 3 5 5 6 2 2 9 4 9 5 8 5 2 8 2 10 5 4 2 2 6 3 7 2 2 9 5 5 ' 9 5 3 7 8 3 4 6 3 1 4 0 4 8 1 7 3 1 9 8 2 10 5 2 8 1 6 8 2 4 4 1 4 5 4 7 1 3 9 3 3 12 11 3 3 9 1 6 8 1 7 2 5 2 3 5 2 9 3 1 9 4 4 10 12 4 4 7 1 12 2 2 7 1 4 1 5 5 6 3 2 8 4 5 9 13 3 3 9 1 9 5 1 6 3 5 2 3 8 3 5 4 3 6 3 8 7 14 7 1 7 8 3 4 8 1 1 5 1 2 11 1 4 7 2 4 11 3 4 15 4 2 9 2 5 3 0 5 5 4 1 5 5 2 9 1 4 8 5 3 12 I I I groups with prior gambling experience stimulus n 15 n 156 1 n = n 6 n = 15 n = 17 n 16 # W L S W L S W L S W L S W L S L S W L S 1 6 6 3 1 3 11 1 6 2 0 3 2 7 6 1 8 8 5 5 8 2 9 1 5 6 1 8 5 1 3 2 0 4 2 3 10 9 6 2 4 3 9 3 5 4 6 0 7 18 2 4 3 1 0 5 2 5 8 2 5 10 1 5 12 4 3 7 5 0 10 5 6 2 0 2 4 6 6 0 13 4 0 9 7 5 6 6 1 2 12 1 5 5 2 0 4 5 6 4 1 4 12 1 3 12 6 6 2 7 2 1 12 4 0 5 2 0 4 4 2 9 1 6 10 2 4 10 7 7 4 4 3 7 5 3 5 1 2 1 3 5 1 7 4 11 2 3 7 6 8 7 2 6. 3 4 8 6 2 1 2 0 4 8 1 6 7 6 4 5 3 8 9 11 1 3 8 4 3 6 0 3 2 3 1 8 2 5 11 3 5 7 4 5 10 5 1 9 7 3 5 4 3 2 3 2 1 4 1 10 5 5 7 2 6 8 11 5 3 7 7 3 5 3 5 1 2 1 3 7 1 7 3 7 7 2 7 7 12 7 4 4 3 7 5 4 5 0 3 2 1 8 3 4 2 11 4 1 10 5 13 5 3 7 ' 8 4 3 4 2 2 3 1 6 4 5 3 10 4 4 6 6 14 10 0 5 8 4 3 6 3 0 3 3 0 9 2 4 10 5 2 8 4 4 15 5 7 3 6 6 3 1 5 2 0 4 4 4 7 2 4 11 2 4 10

of the three-choice alternatives is 11.0 (= 1.5 + 5.0 + 4.5), and thus, under assumption of EV orientation, the choice probabilities of alternatives W, L, and S are 1.5/11.0 =.136, 5o0/11.0 =.455, and 4.5/11.0 =.409, respectively. For convenience, the choice probabilities have been converted into logarithms in the right half of Table 5. As in the previous examples, we again assume independence of observations, so that the likelihood of the whole set of data (observed choice frequencies) or of parts thereof is equal to the product of choice probabilities under assumption of the various models. In logarithms, this means multiplying the choice frequencies from Table 4 to the logarithms of choice probabilities from Table 5, and then summing up over alternatives and stimuli for each model. The antilog of this sum is the likelihood of the data set under the specified hypothesis or model. These likelihoods can be compared pairwise between models (but only for the same data set); however, the resulting likelihood ratios can be compared between data sets, i.e., between the different experimental groups. For some of Hommers' (1973) data, this has been done in Tables 6-9. The sume in the bottom rows are the logarithms of the likelihoods (probabilities) of the respective data, assuming that the probabilities of individual choices are generated by the models named on top of the columns. Of course, they are all negative; the larger their absolute value, the smaller the probability of the data under the respective model. In the order of their likelihoods, we get from the four groups analyzed the following likelihood ratios between pairs of models (see Table 10). 16

TCble 5: Choice pr-obebities from probabilistic choice models BTL choice probabilities assuming logerithms of choice probbilities focussing P focussing V focussing EV focussing P fccussir. V focussing L S W L S W L S W L S W J S A L.067.333.600.500.333.167.16.455.09 -1.179 -.-776 -.2218 -.n010 -.47' -.7t -.:' 5 -.3-2C.200.35-3.-7.583.250. 17.2C0 2 - 990 -.4 377 -.337 -.233 -021 -.77'3 -...^^.111.333.556.500.300.200.263 74.2 -.9547 -.477( -.2549 -.3Cio -.5229 -.690C -.5,0 -.32 2.059.411.530.500.333.167.115 59.346 -1.2291 -.3862 -.2757 -.301 -.4776 -.7773 -.33 -.2.111.333.556.467.533.200.189.105.405.9547 -.4776 -.2549 -.3307 -.4776 -.990 -.7235 -.3025.091.273.636.700.200.100.350.3.3 50 -1.040 -.5638 -.1965 -.1549 -.6990 -1.C0o -.559 -.522 -.200..7.500.7.50.33.167.3.6.384.270 -.6990 -.4776 -.3307 -.3010 -.477 -.7773 -.4609 -.115 7.200.333.467.500.286.214.339.322.339 -.6990 -.4776 -.3307 -.3010 -.5436 -.6x96 -.4 98 -.4921.238.333.429.467.333.200.31 3.278 -.6234 -.4776 -.3675 -.3307 -.776 -.6990 -.25 -.4425.177.294.529.500.300.200.312.312.376 -.7520 -.5317 -.2765 -.3010 -.5229 -.6990 -.50-5 -.5058.200.333.467.4(7.333.200.313.374.313 -.6990 -.4776 -.3307 -.3307 -.4776 -.6990 -.5oa5 -.L271.158.369.473.500.333.167.281.438.281 -.8013 -.4330 -.3251 -.3010 -.4776 -.7773 -.5513 -.355.176.294.530.500.333.167.322.*.322 -.7545 -.5317 -.2757 -.3010 -.4776 -.7773 -.4921 -.4 8<.238.333.429.556.333.111.455.381.164 -.6234 -.4776 -.3675 -.2549 -.4776 -.9547 -.320 -.41o.077.231.692.500.333.169.167.333.500 -1.1135 -.6364 -1599 -.3010 -.4776 -.7773 -. 773 -.4h77 ~ P - ~m~ -.5 -. -0 ~ ~' -. -, _.5 -a -.- 'J -. " = ~jj - - - - '- *> - */ v -. c _r, _ _. _ _.v3 _ v F(S) = -4.2988 -4.3452 = (w) (EV) = -5.7?3

Table 6: Data from group V 8 o stli-uls focussing V focussing EV focussing D sW L S W L S W S ch;'ce. 1^l P i choices log P choices log P choices olo P choices log P choices log P choices lo P choices loz? 1 -.3010 1 -.4776 P -.7773 6 -2 -1.739 1 -.770 8 -.2 2 -.2343 0 -. 21 7 -.777 p -.7P 0 -.5220 7 -.552, P -.o000 47 7 -.73 3 5. -.10 2 -.29 -.990 5 -.P-.550 2.' 8? -.. 9G0 5 -_., 7 - -.3',. '.2 o 4. CO 4 -.477 7-.7773 3 -.o393 r -.3 8 - 1.222 -. l2 -.2 4 -.5337 2 -.4776 9 -.*5990 4 -.7235 2 -.3925 9 -.3925 4 -.0547 2 -.4; 0 72 -.25Lo 6 4 -.1549 2 -.6990 9 -1.C000 4 -.4559 2 -.5229 9 - 4559 4 -1.0410 2 -.s5 -.tlo 7 5 -.3lC 3 -.47, 77 7.7773 5 - -.L57 7 3 -. 7 o 5 3 7 -.,3 7 c 5 -.301C 2 -.5436 8 -.6696 5 -.49 2 -.4921 8 -.o98 5 -.090 2 -.47 8 -.33^^ 9 5 -.33C7 3 -.4776 7 -.6990 5 -.4425 3 -.4425 7 -.55(0 5 -.234 3 -.47: 7 —.375 1 5 -.3010 2 -.5229 e -.6990 5 -.505F 2 -.505 8 -.42P 5 -.7520 2 -.5317 8 -.2 5 11 3 -.3307 3 -.4776 9 -.6900 3 -.5045 3 -.4271 9 -.5045 3 -.69~0 -.47; -.337 12 4 -.310 4 -.4776 7 -.7773 4 -.5513 4 -355 7 -.5513 4 -.8013 4 -.330 7 -.321 13 3 -.3c10 3 -.776 9 -.7773 3 -.4921 3 -.4486 9 -.4021 3 -.745 3 -.5317 9 -. 7 14 7 -.25-9 1 -.4776 7 -.9547 7 -.320 1 -.4191 7 -.7 52 -.234 7. 477 75 1; 4 -.3C1C 2 -.4776 9 -.7773 4 -.7773 2 -.4776 9 -.3010 4 -1.1135-. 2 -. oo ^ —~- -i < _ _ _ ------ H o0 -129.7313 -111.644L -17 1. 1 0 log LR (focussinr EV/focussing V) = -111.64'4 - (-129.7313) 18.0869 ->LR L 1.221 * 1018 log LR (focussing P/focussing EV) = -109.6710 - (-111.6444) - 1.9734 ->- L, 94.0 log LR (focussing P/focussing V) - -109.6710 - (-129.7313) = 20.0603 - LR = 1.149 * 1020

,bl1e 7: -ate fro. group S. 8 c focussing V focussing EV_ focucssing P W L S W L S ' choices lov P choices log P choices 1og P choices!co P hc-hces los n -choice s lov P choices log choeces ^s r - 3 -.3013 C -.4776 = -.7773 3 -. 5 -.5 20 5 -.38P3 3 -1.1730 r 4 -.2343 1 -.6021 3.7773 4 -.37r 1 -.5229 3 -.5528 4.J' 4 -.3010 0 -.5229 4 -.6990 4 -.58c0 0 -.3242 4 -.5800 -.0_ L -.7 _ 4 -.3010 1 -.477 3 -.7773 4 -.9393 1 -.2644 3 -.4609 4 -1.2291 1 -.3I'2 3 -.33C7 0 -.477 5 -.6990 3 -.7235 0 -.3925 5 -.3925 3 -.0957 0 -.477i 5 3 -.1549 0 -.6990 5 -1.0000 3 -.559 0 -.5229 5 i-4559 3 -1.04c 0 -.539 3 -.3010 2 -.4776 3 -.7773 3 -.4e69 2 -.4157 3 -.5f,' 3 -. C9 2 -.4776 5 4 -.3010 2 -.5436 2 -.696 4 -.4698 2 -.4921 2 -.4698 4 -.6990 2 -.77 2 1P' 4 -.3307 0 -.4776 4 -.6990 4 -.4425 O -.4425 4 -.5560 4 -.6234 0 -.476 4 3 -.3010 1 -.5229 4 -.6990 3 -.505P 1 -.5058 4 -.4248 3 -.7520 1 -.5317 3 -.3307 2 -.4776 3 -.6990 3 -.5045 2 -.4271 3 -.5045 3 -.6090 2 -.477t 3 4 -.3010 1 -.4775 3 -.7773 4 -.5513 1 -.355 3 -.5513 4 -.8013 1 -.4330 3 3 -.3010 2 -.4776 3 -.7773 3 -.4921 2 -.44F9 3 -.4921 3 -.7545 2 -.5317 5 -.2549 1 -.4776 2 -.9547 5 -.3420 1 -.4191 2 -.7852 5 -.-23 1 - 2 4 -.3010 1 -.4776 3 -.7773 4 -.7773 1 -.4776 3 -.3010 4 -1.1135 1 -.34 3 -62.7101 -61.6352 -'7.1343 log LR (focussing EV/focussing V) = -61.6352 - (-62.7101) = 1.0749 - LR = 11.88 log LR (focussing EV/focussing P) = -61.6352 - (-67.1343) = 5.4991 -- LR = 3.156 * 105 log LR (focussing V/focussing P) = -62.7101 - (-67.1343) = 4.4242 -> LR = 2.656 * 10 2 -.2', 5.2>,^a -.2 ls -.5'75 -.2' x -.*35 -. 251 -. 150 - 1soo

ro o Table 8: Data from group S 12 o focussing V focussing EV focussing D W L S W L S W L S choices log P choices log P choices log P choices log P choices lcg P choices log P choices log P choices lc p D choices log P 5 -.3010 4 -.477S 4 -.7773 5 -.8665 4 -.420 4 -.388 5 -1.1739 4 -.4776 4 -.2218 8 -.2343 0 -.6021 5 -.7773 8 -7 o0 -.5229 5 -.5528 8 -.6990 0 -.4776 5 -.3307 6 -.3010 0 -.5228 7 -.6990 6 -.5800 0 -.3242 7 -.5800 6 -.9547 0 -.4774 7 -.2549 0 -.3010 5 -.4776 8 -.7773 0 -.9393 5 -.2684 8 -.4609 0 -1.2291 5 -.3822 8 -.2757 2 -.33C7 2 -.4776 9 -.6990 2 -.7235 2 -.3925 9 -.3925 2 -.9547 2 -.4776 9 -.2549 2 -.1549 2 -.6990 9 -1.0000 2 -.4559 2 -.5229 9 -.4559 2 -1.0410 2 -.5r38 9 -.1965 2 -.3010 2 -.4776 9 -.7773 2 -.4609 2 -.4157 9 -.5686 2 -.6990 2 -.4776 9 -.3307 2 -.3010 2 -.5436 9 -.6696 2 -.498 2 -.4921 9 -.498 2 -.6990 2 -.4976 9 -.3307 3 -.3307 1 -.4776 9 -.6990 3 -.4425 1 -4425 9 -.5560 3 -.6234 1 -.4776 9 -.3575 -.3010 3 -.5228 9 -.6990 1 -.5058 3 -.5058 9 -.4248 1 -.7520 -.5317 9 -.2765 3 -.337 1 -.4776 9 -.6990 3 -.5045 1 -.4271 9 -.5045 3 -.6990 1 -.477 9 -.330 3 -.3010 2 -.4776 -.777 3 -.5513 2 -.3585 8 -.5513 -.8013 2 -.4330 8 -.3251 -.3010 -.4776 6 -.7773 4 -.4921 3 -.4486 6 -.4921 4 -.7545 3 -.5317 -.2757 7 -.2549 2 -.4776 4 -.9547 7 -.3420 2 -.4191 4 -.7852 7 -.6234 2 -.4776 4 -.375 1 -.3010 4 -.4776 8 -.7773 1 -.7773 4 -.4776 8 -.3010 1 -1.1135 4 -.3 4 8 -.1599 -116.5222 -94.0674 -88.2646 log LR (focussing P/focussing EV) = -88.2(46 - (- 94.0674) = 5.8028 >- LB = 6.35 * 105 log LR (focussing P/focussing V) = -88.2646 - (-116.5222) = 28.2576 -- LR = 1.785 * 1028 log LR (focussing EV/focussing V) = -94.0674 - (-116.5222) = 22.4548 - LR = 2.85 * 1022

Table 9: Data from group S 12 m focussing V focussinr EV fzccussI' ~ W L S W L S W L ch:ices log P choices log P choices log P choices log P choices log P choices loe P choices log P choices lcg? c'; 1 -.3010 8 -.4776 8 -.7773 1 -.8665 8 -.3420 8 -.3c3 1 -1.1739 8 -.4776 9 -.2343 6 -.6021 2 -.7773 9 -.3768 6 -.5229 2 -.5528 9 -.6990 6 -.477 2 -.3010 5 -.5229 10 -. 6990 2 -.5800 5 -.32L2 10 -.5800 2 -.9547 5 -.477 1 0 -.3010 13 -.4776 4 -.7773 0 -.9393 13 -.26e4 -.4(09 0 -1.2291 13 -.'et2 1 -.3307 4 -.4776 12 -.6990 1 -.7235 4 -.3925 12 -.3925 1 -.9547 4 -.417 1 1 -.1549 6 -.990 10 -1.0000 1 - 4559 6 -.5229 10 -.4559 1 -1.0410 6 -.5x3 4 -.3010 11 -.4776 2 -.7773 4 -.4609 11 -.4157 2 -.5686 4 -.6990 11 -.4 -7 -.3010 6 -.5436 4 -.196 7 -.198 6 -.4921 4 -.4698 7 -.6990 6 -.477 -11 -.3307 3 -.4776 3 -.6990 11 _-4425 3 -.4425 3 -.5560 11 -.6234 -.376 RC 5 -.3010 5 -.5229 7 -.6990 5 -.5058 5 -.5058 7 -.4248 5 -.7520 5 -.5. 3 -.3307 7 -.4776 7 -.6990 3 -.5 5 7.4271 -.5045 -.6990 7 -477 2 -.3010 1 -.477S 4 -.7773 2 -.5513 11 -.3585 4 -.5513 2 -.8013 11 -.-330 3 -.3010 10 -.4776 4 -.7773 3 -.4921 10 -.4486 4 -.4921 -.7545 10 -.53 1 10 -.2549 5 -.4776 2 -.9547 10 -.3420 5 -.4191 2 -.7852 10 -.6254 5 -.477 2 -.3010 4 -.4776 11 -.7773 2 -.7773 4 -.4776 11 -.3010 2 -1.1135 4. 4 -139.2482 -112.0598 -117.4016 log LR (focussing EV/focussing P) = -112.0598 - (-117.4016) - 5.3418 rR = 2.197 * 105 log LR (focussing EV/focussing V) = -112.0598 - (-159.2482) = 27.1884 -- LR = 1.543 * 1027 log LR (focussing P/focussing V) = -117.4016 - (-139.2482) = 21.8466 -> LR = 7.024 * 1021 -ices lo P C -.2218 2 -.3307.0 -.2549 4 -.2757.2 -.2549 LO -.19 5 2 -.3357 - -.33C7 3 -.375 7 -.2',r5 7 -.3307 -.3251 4 -.2757 2 -.34575 1 -.1599 J

Table 10: Examples of likelihood ratios from Hommer's data Group 8 V o (8-year-old normal students without gambling experience): likelihood ratio more favored modelbetween: focussing P focussing EV less favored model-focussing EV 94.06 rank order of models: focussing V 1.149 * 1020 1.221 * 1018 P - EV - V Group 8 S o (8-year-old educable retarded children without gambling experience): likelihood ratio more favored modelbetween: focussing EV focussing V less favored model-focussing V 11.88 rank order of models: focussing P 5.156 * 105 2.656 * 104 EV - V - P Group 12 S o (12-year-old educable retarded children without gambling experience): likelihood ratio more favored modelbetween: focussing P focussing EV less favored model-focussing EV 6.35 * 105 rank order of models: focussing V 1.785 * 1028 2.85 * 1022 P - EV - V Group 12 S m (12-year-old educable retarded children with gambling experience): likelihood ratio more favored modelbetween: focussing EV focussing P less favored model-focussing P 2.197 * 105 rank order of models: focussing V 1.545 * 1027 7.024 * 1021 EV - P - V,,,...,.. ~

Similar analysis could be performed for other 10 of Hommers' 14 groups too. We have displayed in the rightmost column of Table 10 the rank order of models as indicated by the likelihood ratios calculated from the data; although the likelihood ratios themselves differ considerably, it is interesting to note that 12 year old retarded children show the same rank order of models as the 8 year old normal children, thus supporting Hommers' hypothesis of retardation as a shift in development. Also, comparison of the results from 12 year old educable retarded children without gambling experience with those from their classmates with prior gambling experience unveils a considerable influence of this experience on choices among gambles. Besides these analyses for individual groups, larger groups can be taken into consideration, e.g., likelihood ratios between models can be calculated over all Ss with prior gambling experience, or over all retarded children to be compared to those calculated over all normal children, etc. Since we used these data only for illustrative purposes, we need not go into further detail. Also, we will turn to the problem of interpretation of such analyses later in this paper 23

Seghers, Fryback & Goodman's Data The next set of data we are going to use are those of Seghers, Fryback & Goodman (1973). They presented their Ss sets of 7 gambles, like those reproduced in Table 11: Table 11: List #1 as an example bet # 1 2 3 4 5 6 7 win on 4 1.55 3.45 5.50 7.15 8.95 10.80 12.65 lose on 32 1.10 1.15 1.20 1.25 1.30 1.35 1.40. EV -.806 -.639 -.478 -.317 -.162 0 +.162 - Var.683 2.088 4.469 6.963 10.423 14.567 19.479 Wins and losses were determined by means of a roulette wheel which was respun if 0 or 00 occurred, such that "win on 4" (numbers) mean? a winning probability of 4/36 = 1/9, etc. Seghers, Fryback & Goodman's lists varied in (1) expected value (EV), (2) range of outcomes (A-B), (3) step size of expectation increase (AEV), (4) position of the maximal EV bet (OBP). Dependent variables were: (a) choice of most perferred gamble. (b) rank orderings of the sets of 7 gambles. 24

Although the experimental design looks as though a factorial design AVOVA had been planned, the data don't permit - uehh an analysis. A frequency analysis as suggeste, by Sutcliffe (1)57) would be more appropriate, however, low expc,'te,,e-(1 freqlluenli(s in thel overall contingency table prohibits such an analysis. A Bayesian data analysis is suggested as an alternative. However, since Seghers, Fryback & Goodman assume a deterministic decision making model, this analysis runs into the problems mentioned before. The simple probaoilistic choice model used to analyze Hommers' data is no longer appropriate here since there are negative expectations which are not compatible with a BTL choice model based on these expectations as scale values. Deterministic decision making models predict choice of the optimal gamble with probability 1, and of all other alternatives with probability 0 if gj is optimal P(choice of gamble gj) = a Lo else where "optimal" is defined in the context of the respective decision making model to be tested, e.g., it would be the maximum EV bet under the expectation maximization model, or the ideal risk bet under assumption of Coombs Portifolio Theory. Unfortunately, likelihoods of 0 or 1 cannot be handled by the Bayesian data analysis model. Thus, we have to modify these models somehow to get away from the 0-1 likelihoods. There are several ways to do so of which we will try to (1) keep the deterministic model in principle, but dilute the too peaked o-l likelihood function by allowing for some error variance, 25

(2) modify the deterministic hypothesis somewhat arbitrarily to smooth its peak, following an example given by Pitz (1968), who encountered a similar problem, (3) abandon the deterministic model completely in favor of some probabilistic choice model (as they have been used for riskless choices for a long time), (4) replace the deterministic model by some hybrid of deterministic and probabilistic components. We will explore all these possibilities in turn. (1): Introducing error variance: Our suggestion is to dilute the too peaked likelihood functions somewhat by allowing for error variance: The diluted H no longer assumes Ss always pick the maximal EV gamble, but rather assumes that Ss err sometimes in the sense that they don't choose a certain gamble although they mean to choose it. Fortunately, the data by Seghers, Fryback & Goodman provide a way to estimate these error rates: they had their Ss do the task twice. Our suggestion is to use the observed discrepancies between first and second choice (under otherwise equal conditions) as estimates of error rates. To do so, the Ss first and second choices of gambles are tallied in 7x7 confusion matrices, separately for each given position of optimal EV bet (OBP). A completely consistent S should make the same choice on both occasions: i.eo, all entries should be in the main diagonal, and all other cells should be empty. Every deviation from this diagonal matrix is considered an "error," an inconsistency, a deviation of the S from his pure strategy assumed under the hypothesis of 26

expectation maximization, H. Assuming that Ss err at both choices, i.e., both 1st and 2nd choices have a chance to deviate from the Ss' true choice predicted by his strategy, we take the average of row anid co'lumn distribution for each stimulus as its error distribution. This procedure assumes that, on the 2 days, S at least once chooses his "ideal bet" without making an error. It does not take into account those cases where S "wants to" select a certain bet but "misses" on both days, This may lead to an underestimation of error rates. A better way would be to get confusion probability estimates from more often repeated choices, in a complete pair comparison matrix, or from a different task, like the procedure used in DeSoto & Bosley (1962) (quoted in Coombs, Dawes & Tversky, 1970, p. 68 ff.). This cannot be done with these data, but it could be in future experimentsif you want to make the assumption that confusion of memory traces is representative of confusion in choices. Now, with this knowledge about S's error probabilities, we can modify the 0-1 distribution under the former pure expectation maximization hypothesis: We diminish the peak of the distribution (formerly P(D|H1) = 1 at maximal EV bet) by replacing the 1 by the repetition rate (1st choice =2nd choice) in 1st choice/2nd choice confusion matrix, and by replacing the zeroes by the relative frequencies with which Ss have chosen the respective gambles "erroneously." Thus, the EV maximization hypothesis H implies data probabilities of 1 P(D |H ) = the repetition probability of the maximal EV bet for o 1 the maximal EV bet (D ) chosen o 27

and P(D H ) = the probability of choosing Di given S has chosen D on io the same trial in the 1st or 2nd repetition. the same trial in the lst or 2nd repetition. (~ P(Di|Hl) should be 1 if everything is correct.) Analogous computations can i be done for other alternative hypotheses, like variance perference, alsoo Tables 12 and 13 give examples of such confusion matrices between 1st and 2nd choice: Table 12 are absolute frequencies; Table 5 is the same matrix with a matrix of ones added to it. (Actually, the entries in Table 12 are averaged over 2 presentations.) The rationale for adding these ones to the cells is again a Bayesian one: we are revising here, in principle, Dirichlet distributions (see, e.g., Novick & Grizzle, 1965). We start with a uniform (flat) prior distribution D(1, 1, 1, 1, 1 1, 1) with all parameters equal to 1, and then add to them the numbers of observations to obtain the parameters of the posterior distribution after Bayesian revision. However, summing cell entries from row and column would assume independence of observations from the two sessions which probably is not given since we assume that S's choices were influenced by the same preference structure on both days. Thus, to avoid an overly peaked Dirichlet distribution, we average over column and row entry rather than adding them up. Actually, this does not make a difference as long as we calculate only means and not variances. 28

Table 12: Choice on day 2/cioice on day 1 averaged confusion matrix G + R10 2 v s 1 2 4 5 6 7 Overall opt. bets 1 2 5 4 5 6 7 116.5 13.5 9.5 5 5.5 1 8 12.5 135.5 7 1 1 1 2.5 7.5 9.-5 25.5 7.5 1 2 5 2.5 6 6 6 6.5 2.5 1.5 1.5 2.5 2 16.5 5-5 5.5 3 5 0 1.5 0.5 3 2 2.5 1 3 0.5 4 1.5 37 148.5 45.5 54 26 32.5 14 63.5 I 157 58.5 58 33.5 32.5 8.5 56 384 Table 13: Matrix with 1 added to every cell 1 2 4 5 6 7 + L in a iI cells 1 2 5 4 5 6 7 117.5 14.5 10.5 6 4.5 2 9 13.5 14.5 8 2 2 2 5.5 8.5 10.5 26.5 8.5 2 5.5 2.5 7 2.5 7 3.5 7 3 7.5 17.5 5.5 4 4 1 1.5 3 2 1.5 2.5 6 2.5 4 5.5 4 5 58 155.5 52.5 61 5395 59.5 21 70.5 6 5 6.5 4 - 164 45.5 65 40.5 39.5 15.5 63 433 29

As an illustration, assuming that gamble #1 is the optimal bet in the Ss' view (H2), and having observed the number of choices displayed in Table 13, we get: Table 14 from row 1: 117.5 13.5 8.5 3.5 2.5 4 6 from column 1: 117.5 14.5 10.5 6 4.5 2 9 sum of both: 235 28 19 9.5 7 6 15 average: 117.5 14 9.5 4.25 3.5 3 7.5 and thus the choice probabilities:.734.088.060.027.022.019.047 for gamble#: 1 2 3 4 5 6 7 when gamble #1 is the "true choice" assumed by the model. Some results of such tallies are reproduced in Table 15, assuming various choice strategies on the side of the Ss. Column 2 displays choice probabilities under an a priori random-choice null hypothesis (all gambles chosen with equal probability 1/7 =.143). Table 15 (1) (2) (3) (4) (5) (6) (7) (8) (9) 10 ed H1: maximize EV: > ~ 1> - a~ cr, ~ maximal EV is in gamble rA+ A+ c~i *^ *....o co c-c co w 0O0 #1 #3 #5 #7 x gz H, LC V.,_, PI P PA 1.143.802.110.080.092.734.112.127 2.143.060.140.051.040.088.818.054.116 3.143.038.566.058.046.o6oJ.o80 4.143.031.065.124.050.030.063.117.594 5.143.019.024.482.040.022.051J 6.143.018.035.082.062.01.025.652.062 7.143.032.060.117.670.047.057J.105.~~ ~ ~~~~~~~~~ 04 0 57. 1 05 30

Columns 3 through 6 are the diluted choice probabilities assuming expectation maximization with some errors, calculated in the manner described above from confusion matrices between choices in first and second sessions of Ss but tallied separately for lists where gambles 1, 3, 5, and 7 were optimal, respectively. Column 7 is calculated from the tallies illustrated in Tables 12, 13, and 14, assuming that Ss have the strategy of always picking gamble #1, no matter what the parameters of the gambles in the list are. Columns 8 through 10 are choice probabilities calculated under similar hypotheses, assuming that Ss have preferences for certain regions of the lists of gambles presented to them, i.e., that they always pick gambles #1-3, or #5-7, or #3-5, respectively. With the choice probabilities from Table 15 taken as P(DIH.), all these models can be tested against each other by calculating the respective likelihood ratios. To make the analysis more convenient, all hypotheses could be tested first against the random-choice null hypothesis (H ). The resulting likelihood ratios against H could then be divided by each other to yield likelihood ratios agains each other since P(DIHi) / P(DIH.) P(DIHi) P(D| H) P(DJH0) P(D|H ) However, this is only feasible as far as Hi and H. are mutually exclusive. H H and H in Table 15 are not since they all assume a strategy to choose gamble #lo

The choice probabilities assumed under hypotheses Hi through H5 from Table 15 yield the likelihood ratios reproduced in Table 16 if tested against the uniform distribution H o To use Table 16, we multiply the entries by the prior odds every time the respective datum comes up; e.g., to test hypothesis H against H, we would 1 0 multiply prior odds (i.e., odds so far obtained) by 5.14 if S chooses gamble #1, and gamble #1 is optimal (maximal EV) in the respective list. Table 16: Likelihood ratios calculated from Table 15 (1) (2) (3) (4) (5) (6) (7) (8) (9) Gamble LR/ Ga e 1 opt pt l/o 7 pt LR2/0 LR3/0 R4/0 LR5/o _ 1 opt 3 opt 5 opt 7?opt 1 5.61.77.56.64 5.14.78.89 2.42.98.36.28.62 1.91.38.81 35.27 3.96.41.32.42.561 4.22.46.87.35.21.44.82 1.39 5.13.17 3.37.28.15.221 J 6.13.25.57.43.13.18 1.19.43 7.22.42.82 4.69.33.40J.74 32

Again, it will be more convenient to do this in terms of logarithms, thus we have, in Table 17, the log LR /O in column 3, and the number of choices for the respective gamble in column 2. Table 17 -T(1) (-(2) (3) (4) (5) gamble number of log log log ~# _ choices LR1/0 LR2/0 LR4/ 1 3 -.1938 +.7110 -.1079 2 0 -.5528 -.2076 -.4202 3 2 -.4949 -.3768 -.2518 4 2 -.4559 -.6778 -.0862 5 1 -.5528 -.8239 +.0755 6 1 -.3665 -.8861 +.0755 7 15 +.6712 -.4815 +.0755 log LR +6.6657 - 8.9087 +.2838,R 4.6531*106 1/(8.104*108) 1.922 The data in column 2 are the choices made by 12 Ss in 2 sessions among the gambles of list #1, reproduced in Table 11, where gamble #7 had maximal EV, such that the logarithms in column 3 of Table 17 are those of the likelihood ratios in column 5 or Table 16. The sum of the products of entries in columns 2 and 3 of Table 17, the overall log likelihood ratio, is 6.6657, indicating a likelihood ratio of 4.631*10 in favor of expectation maximization (H ) over random choice (H ). 1. o Columns 4 and 5 show the respective log LR for hypothesis H2 (always pick gamble #1) over the random choice hypothesis H, and for hypothesis H4 (always pick gamble # 5, 6, or 7) against the random choice hypothesis H. Resulting likelihood ratios LRo = 8 4 8 in favor of H (random choice) over H li / 28. o 2 33

(always pick gamble #1) with these data, and LR4/ = 1.922 in favor of H4 (always pick # 5, 6, or 7) over H (random choice). So far, we have analyzed only the choices among gambles of one list- of course, it is feasible and advisible to do it over the whole set of data from all lists, simply by summing up the respective log LR 1/ over all data for the various hypotheses Hi. Seghers, Fryback & Goodman have done this for each of their Ss, individually, and we are reproducing their results for one of their Ss as an example in Table 18. Besides calculating likelihood ratios LR1/0 for the aforementioned hypotheses H. against the random choice hypothesis H over 1 o all (lists) (column 2), they also did it for specified subsets of lists, e.g., lists with high EV (column 2), lists with low EV (column 4), lists with high EV differences between gambles in the lists (column 5), lists with low EV differences (column 6), lists of gambles with large variances (range of bet, i.e., Iwin-lossl) (column 7), and lists of gambles with small variances (column 8). Thus, it is possible to compare data likelihood, for the various hypotheses H under different stimulus conditions. This breaking down likelihood ratio analyses into analyses over mutually exclusive subsets of the whole data set corresponds roughly to what is done to the sum of squares in analysis of variance (ANOVA), or to the chi square in analyses of multi-dimensional contingency tables (e.g., see Sutcliffe, 1957): It shows how much the respective subsets of data (i.e., data under specific conditions) contribute to the overall likelihood ratio. To make fair comparisons of this kind, we have to take care that these subsets are of equal sizeo 34

Table 18: Likelihood ratios for S #1 of Seghers, Fryback & Goodman (1) (2) (3) (4) (5) (6) (7) LR calculated over: competing only high only low only high EV only low EV only large or hypotheses all lists _ hypctheses ____ all, liss EV lists EV lists difference lists difference lists range lists ra: LR o: expectation (EV) maximiz expectation (d ) mcaxice 2867.13 33-5 85.6 262.2 10.9 46.6 mization vs. random choice LR2/ ~ always pick #1 vs. LR2/0: lways pick #1 vs. 3715.4 2752.6 1.4 109.7 33-9 3615l1.3 random choice LR3/o: always pick #1,2,3 622.2 1356. 4.6 66.6 9.5 302.8 vs. random choice LR4/o: always pick #5,6,7 1/19743.6 262.2 75.3 170.8 115.6 62.2 vs. random choice LR5/o: always pick #3,4,5 vLR/: ralways pick #e,4,5 1/2.7 1/7.9 2.9 1/2.9 1.1 1/3.2 vs. random choice Note: reciprocal values (l/x) indicate that the data were, in these cases, more likely under Ho than under Hi (8) ily s.all nge lists 6.4 1/97.3 2.1 317.2 1.2

The product of the likelihood ratios LRi/j competing hypotheses Hi, Hj from exhaustive and mutually exclusive subsets of data equals their likelihood ratio over the whole data set. E.g., in each row of Table 18, the products of entries in columns 3 and 4, 5 and 6, or 7 and 8 equal each other, and equal the entry of column 2, except for rounding errors. (This provides, by the way, an easy means of checking computations.) The results of such likelihood ratio analyses over the subsets of data can be used to find out under which conditions which hypotheses are how much more likely than others, and thus may lead to more specific theories about the underlying pattern of behavior. The comparison of likelihood ratio analysis to more conventional methods like ANOVA is not always straightforward; the easiest comparable traditional technique would be a frequency analysis because it deals with the frequencies of occurrence of events which enter directly the likelihood ratio analysis (as exponents.) Seghers, Fryback & Goodman did analyses of variance over the same data we used for demonstration in Table 18, both terms of absolute deviation of bet number as dependent variable, and in terms of absolute deviation of bet number as dependent variable, and in terms of absolute deviation of bet number chosen from maximal EV bet number in the respective list. Results (for the same S, and same session as in Table 18) are shown in Table 19. Seghers, Fryback & Goodman's lists were constructed in such a way that, given the maximal EV bet in the list (in positions #1, #3, #5, or #7 of the list = optimal bet position OBP), the adjacent gambles decreased in EV to both

Table 19: Analyses of variance for choices of S #1 of Seghers, Fryback & Goodmgn ANO7A of absolute deviation of bet ANOVA of absolute number hosen from axmf be chosen from aximal EV betf cosen source of variation df - -- 2 mean F-rato mean F-rat.o I variance nmean - e scuare if > i accoui;ted for sq are if > 1 accc-:te for -, maximal EV (EV) EV difference (DEV) range (R) optimal bet portion (OBP) interactions: EV x DEV EV x R EV x OBP DEV x R DEV x OBP R x OBP EV x DEV x R EV x DEV x OBP EV x R x OBP DEV x R x OBP residual (error) 1 0 0 2.000 2.000 1 2.000 2.000 0 0 1 0.125 0.125 3 45.625 15.208 5.125 3.125 20.125 6.708 4.31 1.55 6,-0 1 1.125 1 0 3 4.750 1 4.500 3 5.750 3 10.625 1 3.125 3 32.625 3 6.750 3 3.250 J 10.625 1.125 0 1.583 4.500 1.917 3.542 3.125 10.875 2.250 1.083 3.542 1.27 3.07 16% 6.125 2 4.250 0.500 8.250 10.125 0.125 28.125 5.250 6.750 13.125 109.875 6.125 2 1.417 0.500 2.750 3.575 0.125 9.575 1.750 2.250.5 375 1.40 2.14 total 31 130.875

sides by a step size DEV = difference in expected value, Thus, the dependent variable "absolute deviation of number bet chosen from number of maximal EV bet" can be considered a measure of S's deviation from expectation maximation behavior. Whereas such independent variables like "high level of maximal EV in list" versus "low level of maximal EV in list" (first line in Table 19), large step size of EV differences in list versus small step size (line 2 in Table 19), and range of outcomes of gambles (line 3 in Table 19) show no significant difference in the dependent variables, there are some differences between the contributions of the respective subsets of data to the likelihood ratio between expectation maximization and random choice hypotheses in Table 18 (line 1). However, we have no means to compare these two kinds of analyses quantitatively. Testing the various hypotheses Hi about choice behavior against the random choice hypothesis H is the approach to their evaluation that comes closest to 0 traditional hypothesis testing. Testing them against the most descriptive choice probabilities is another possibility these likelihood analyses offer for which no counterpart exists in traditional statistics, Comparisons of data likelihoods under the various hypotheses aforementioned to these (by definition) maximal likelihoods can show how far out hypotheses Hi deviate from actual behavior. These most descriptive choice probabilities specify upper bounds for data likelihoods, under the choice hypotheses, as illustrated in Figure 1. 38

L(D) I Ho1 normative models H optimal description random choice, uniform p maximum likelihood p Figure 1 The most descriptive (maximum likelihood) vector of choice probabilities for the seven gambles can be obtained for each subject from his choices by the following method: the data-choices of one out of seven gambles in each list-are generated by a multinomial distribution, with choice probabilities p. following a Dirichlet distribution. Thus we can assume a flat Dirichlet distribution D(1, 1, 1, 1, 1, 1, 1) as prior, a multinomial data generating process yielding xj choices of gamble gi, and thus leading (via a Bayesian probability distribution revision) to a Dirichlet posterior distribution, D(1 + 1, x2 + 1, 5 +1 x4 + 1, x5 + 1 6 + 1, x7 +1). This Dirichlet posterior distribution gives us the probability P(plx) of vector of choice probabilities (P', P2, P 3, Pk, P5, P6, P7) = p of gambles gl through g7, given the vector of observed choice frequencies (xl, x2, 3, x4, X5, x6, x7) = x, and what we need is that vector p for which P(plx) is maximal over the space of all possible po (Note that this space is restricted by ESp = 1 for each p.) J We take S #1 of Seghers, Fryback & Goodman, again, as an example. His (or, rather, her) choices are reproduced in columns 2, 5, 8, and 11 for the respective OBP conditions, and summed up in column 14 of Table 20. Columns 3, 6, 9, and 12 contain the choice probabilities under the diluted expectation maximization hypothesis H from Table 15, in columns 4, 7, 10, and 13 we find correspon logarit. The log ikelihood for expectation maximization t,~c corresponding logarithms. The log likelihood for expectation maximization 39

Table 20 (1) (2) (3). (4) (5) (6) (7) (e) (9) (10) (11) (12) (15) (1L) (15) (16) (!) (!c) (lo) choices of S 1l when optimal gamble was (expectation maximation hypothesis (H1) maxl d r:tive choe ( maximal descriptive strrte ' (H )gertme.rnble ge 1 _ gmble 3 gam.ble 5 gamble 7 __ # choice choice choice choice total # 4 of S's - choice choices prob. log P choices prob. log P choices prob. log P choices prob. log P of choices choice lo c? lor (HI)(H1) (H1) (H1) choices +1 prob. 1 13.802 -.0958 7.110 -.9586 3.080 -1.0969 5.092 -1.03, 2 28 29.4OO -.35 3.143 -,Lh47 2 1.0(0 -1.2218 4.14O -.8539 3.051 -1.2924 4.040 -1.3979 12 13.185 -.7375.143 -.PL7 3 1.038 -1.4202 2.566 -.2472 3.058 -1.2366 2.o46 -1.3372 8 9.12, -.^o 2.143 -.I L7 4 1.051 -1.5086 3.0'5 -1.1871 3.124 -.9066 0.050 -1.3010 7 8.133 -.*.1 3 -.Q447 5 0.019 -1.7212 0.024 -1.6198 4.482 -.3170 0.040 -1.3979 4 5.C0O -1.1514.1l3 -.P4,' 6 0.018 -1.7447 0.035 -1.4559 0.082 -1.0862 1.062 -1.2072 1 2.028 -1.5528.143 -.P, 7 0.032 -1.4949 0.0O0 -1.2218 0.117 -.9318 4.670 -.1739 4 5.070 -1.15?.143 -.8447 log L 64 71 -49.7932 -4.3 123 -54. C 08 0

calculated from these figures is -49.7932. The S's most descriptive strategy, computed as outlined in the preceeding paragraph, is given in column 16, with the corresponding logarithms in column 17. The log likelihood from these figures (which is tile maximal attainable) is -44.3125, and the log likelihood of this S's choices under the random choice hypothesis H is -~~~~~~~~- ~~O b4 * log 1/7 = -54.0608. The expectation maximization hypothesis (H ) comes much closer to the subjects most descriptive strategy (H7) than to the raniom choice strategy (H ). The respective likelihood ratios are o 5 4 Li /l = 5.026 * 10 l L/0 = 1.852 * 10 and LR7 5.604 * 109 We have so far used the assumption that Ss occasionally deviate from their ideal choice and make "errors" in their decisions which we could use to get rid of the choice probabilities of 0 and 1 assumed by the deterministic normative models of decision making. 41

Expectation Preference Model In discussing Hommers' paper, we have seen that the assumption of probabilistic preference models rather than deterministic choice models is another feasible way to avoid choice probabilities of 0 and 1. For gambles of the form g. = (w, p., 1.) where Ss wins the payoff wj with probability pj and loses 1. with probability (l-pj), this model assumes that Ss choose a gamble gj with probability P(gj) proportional to the relative J 3 utility U(g.) of the gamble gj, J P(g.) = U(g.)/E U(gj), J J. a J where U(gj) = EV(gj) = pw. + (l-p)l under the expectation preference model. For each choice of g an S makes, J P(gj) is the likelihood of this observation to occur under assumption of this model. This expectation preference model works fairly well for sets of gambles where all EVs are positive, as we have seen in the analysis of Hommers' data. However, it will run into difficulties if the EV of one or more gambles in the list (set of choice alternatives) is negative or zero. A Thurstonean (rather than Lucean) choice model might help in this case. Here, choice probabilities are only dependent on differences between utilities

of choice alternatives, and not on their absolute values. Under the assumptions ot' this model, the probability of choosing one element (i.e., a gamble) in a pair of alternatives is equal to the integral of the normal distribution from - oo to the difference in utilities (expected values) of the respective pair, where the mean of this normal distribution is 0, and its variance is the variance of the utility difference which is the sum of the variances of the discriminal dispersions of the two elements (gambles) in the pair, if we assume independence (uncorrelatedness) of these two discriminal processes. Application of this model requires estimation of these variances which can be obtained from repeated choices. Regret Avoidance Models A way to apply a Lucean choice model to choices among bets including gambles with EV < 0 might be to consider regrets rather than payoffs. Regrets are obtained from payoffs by reducing them by the maximal amount obtainable with each given state of world. Regrets calculated by this method are all negative; they are measures of undesirability rather than desirability. Thus, it does not make sense to assume choice probabilities proportional to regrets. What we need is some antitone transformation on the regrets which leads to high choice probabilities for low regrets, and low choice probabilities for large regrets. We propose three simple models for this purpose: (a) the sum-difference regret model assumes that choice probabilities are proportional to the deviation of the respective expected regrets from the sum of all regrets, 43

i ri- ri P(i) (N- 1) r th where ri is the expected regret associated with the i alternative, smallest regret being O, N=number of alternatives. Model (a) gives choice probabilities with a rather small variance, i.e., the choice probabilities are not very sensitive to differences in regrets. (b) the reciprocal regret model assumes that choice probabilities are proportional to the reciprocals of the respective expected regrets, P(i) = rZl i ri This leaves P(i) forri = 0 undefined. Model (b) leads to stronger deviations of choice probabilities from a uniform distribution over alternatives to differences in regrets, i.e., model (b) is more sensitive, but cannot always be used because if leaves the choice probability for an expected regret = 0 undefined. (c) the max-difference model assumes that choice probabilities for alternatives i are proportional to the differences between the respective expected regrets and the maximal expected regret, max [r] - ri P(i) - N max [r.] - Z r 1 1 i=l i This model is more sensitive to differences in expected regrets than model (a) and leaves no choice probabilities undefined as does model (b), but leads to a 0 choice probability for the maximal expected regret alternative. This is an undesirable consequence for a BTL choice model but may be quite 44

realistic. In the data analysis, it will hurt only if any S picks the maximum expected regret gamble. For the example of list #1 from Seghers, Fryback & Goodman (see Table 11), Table 21 shows the respective choice probabilities with these probabilistic regret avoidance models in columns 8, 11, and 14, with the corresponding logarithms in columns 9, 12, and 15. Column 17 displays the choice probabilities under error-diluted deterministic expectation maximization hypothesis H as given in Table 15, and column 18 of Table 21 contains their logarithms. In column 19, we have the actual numbers of choices made by S in this list of gambles, for which we calculated the likelihoods under the hypothesis H (random choice), H (diluted expectation maximation), H8 (reciprocal regret), H (sum-difference regret), and H (max-difference regret). Table 22 displays 9 10 the pairwise likelihood ratios between these hypotheses. As we can see, the data are 1067 times more likely under the diluted deterministic expectation maximization hypothesis H than under the most favored probabilistic regret-avoidance hypothesis H8. The data likelihood under the least favored probabilistic regret-avoidance hypothesis H is almost as large 9 as under random choice assumption Ho, LR9/0 = 1.111. This indicates that for likelihood ratio analyses of choices among bets made by adult subjects, error-diluted deterministic expectation maximization models seem much more likely than probabilistic preference models. However, in the case of Hommers' data where no source to estimate the error rate was available, probabilistic preference models proved quite useful. It should be mentioned that neither of these studies was originally designed for a likelihood ratio analysis-if this had been the case, adequate measures would 45

Table 21 (1) (2) (3) (4 f5~ ( ') (7) (p) (9) (10) (11) (12) (13) (14) (15) (16) (17) (18) reciprocal regret model sum-diff. regret model max.-diff. regret model expectation max. payoffs expected (H) (H9) (H) model (H)er of g8 r,-,b 1 eHexpected 9 10 number of gsa ble -------— regrets regret choice choice choice choices win loss (ER) 1/ER prob. log P FER-ER prob. log P ERx prob. log P log P made by S win loss ER-ER___ (H^) (H) ^"^ (H^) ______ prob> ________________p (He) (H) (H10 i 1.55 -1.10 11.10 0 1.22.8197.069 -1.1612 3.99.096' -1.0177 0 0 - 0.092 -1.0362 3 2 35.45 -1.15 9.20.05 1.O6.9434.079 -1.1024 4.15.133 -.8761.16.048 -1.3188.040 -1.3979 0 3 5.30 -1.20 7.35.10.90 1.111.093 -1.0315 4.31.138 -.8601.32.096 -1.0177.046 -1.3372 2 4 7.15 -1.25 5.50.15.74 1.3514.113 -.9469 4.47.143 -.8447.48.144 -.8416.050 -1.3010 2 5 5.95 -1.30 3.70..59 1.6949.142 -.8477 4.62.148 -.8297.64.189 -.7235.040 -1.3979 1 6 10.80 -1.35 1.85.29.43 -.3256.195 -.7100 4.78.153 -.8153.79.237 -.6253.062 -1.2076 1 7 12.65 -1.4 0 0.30.27 3.7037.310 -.5086 4.94.158 -.8013.95.284 -.5467.670 -.1739 15 prob. 1/9 8/9 1/9 8/9 r.ats 12.65 -1.10o 1.22 r5.2i 11.9498 31.26 3.33 24 log LH = 24*(-.8447) log i.i -16.6271 -20.1272 -- -13.5990 - -20.2728 (N log LRl/9 = 6.5282 log LRI/e = 3.0281 log LR8/9 - 3.5001 log LRo/l = 6.6738 log LRo/9 = 0.1456 log LRo/8 = 3.6457

Table 22 more favored hypothesis likelihood hypothes i ratio between:ween diluted EV H8: reciprocal H9: sum-diff. maximization regret regret H8: reciprocal 1.067 * 10 regret lyess favored: sum-diff. 75* 106 316 * hypothesis regret ch random i.719 * 10 4.4253 * l0 choice

have been provided beforehand. Pitz, 1968 found another way of handling the problem of data probabilities of 0 and 1, in another context, but also with data originally not observed with a likelihood ratio analysis in mind. He tested a (null-) hypothesis H of equal probability of two kinds of observations (p = 0.5) against the rather unspecific hypothesis H1 of p > 0.5. The data showed that 32 out of 48 Ss gave responses in accordance with H1. The likelihood ratio for these data would have been 48 L 532 16 P (1-Pl) From this equation Pitz determined the value of p for which the data would 48 32 16 be equivocal, i.e., for which L would be one:.5 = (1-p) P1.8. (That means: if H meant p >.8, the data would actually favor 1 H rather than H.) Pitz's suggestion is to consider H1 as a distribution g(p) over p rather than a constant pi, such that the likelihood ratio is 48 L = 1.0 52 16 ~ P (l-p) g(p) dp.5 and he proposes several possible distributions g(p), such as a uniform (rectangular) distribution over [.5, 1.0], a triangular distribution with g(p) = 0 for p <.5, and a kind of beta distribution with a rather high mean. Such an analysis could be done with the Seghers, Fryback & Goodman data, too. 48

Conclusion Now that we have seen that we can figure likelihood ratios between various competing hypotheses on given data sets which were not even made for it, what do we do now? For a complete Bayesian data analysis, we would multiply our computed likelihood ratios to some prior odds for the respective hypotheseso These prior odds may be more or less public, or may be our very personal belief states. Methods to elicit and assess such prior distributions have been introducted and discussed elsewhere (e.g., Winkler, 1967, Stael von Holstein 1970). For a complete Bayesian analysis, we would consider the possible consequences of our decisions between competing hypotheses, in terms of utilities assessed to the various combinations of our decisions among hypotheses with the possible "true" states of the world, and use these utilities in connection with our prior odds to determine cutoffs for the likelihood ratios where to decide in favor of which hypothesis or model. There are various techniques available now for the assessment of utilities to outcomes, even if these outcomes are characterized by several revelant attributes. These techniques have been summarized recently by Fischer (1972). As we have seen in the few examples given in this paper, likelihood ratios grow rather rapidly with larger amounts of data. Even very biassed prior odds would be brought very soon into the correct range by multiplication to These large likelihood ratios. This indicates that Bayesian analyses might get along with much smaller sample sizes than traditional statistical data analyses

with their diffuse alternative hypotheses. How much precisely can be economized on the sample size, will depend in each case on the cutoff determined by prior odds and costs and payoffs (utilities) involved, as indicated by a proper decision analysis (see, e.g., Raiffa, 1969). That a careful formulation of competing hypotheses alone can result in considerable savings on expected sample size, has been shown by Wald (1947) already. 50

References Atkinson, R. C., Bower, G. H., & Crothers, E. J. An introduction to mathematical learning theory. New York: Wiley, 1965. hush, Ro R. Estimation and evaluation. Handbook of Mathematical Psychology, Ch. 8, Vol. 1. Wiley, New York, 1963. Cunningham, D. R., & Briepohl, A. M. Empirical Bayesian learning. IEEE Transactions on Systems, Man, and Cybernetics, SMC-1 (1), 19-23, 1971. Edwards, W., Lindman, H., & Savage, L. F. Bayesian statistical inference for psychological research. Psycholo Rev., 1963, 70, 193-242. Fu, K. S. Sequenti;l methods in pattern recognition and machine learning. Academic Press, New York & London, 1968. Gettys, Co F., & Willke, T. A. The application of Bayes's Theorem when the true data state is unknown. Organizational Behavior and Human Performance, 1969, 4, 125-141. Kuhn, T. S. The structure. of scientific revolutions. University of Chicago Press, 1962. Lindley, D. V. A statistical paradox. Biometrika, 1957, 44, 187-192. McGee, V. E. Principles of statistics: Traditional and Bayesian. AppletonCentury-Crofts, New York, 1971. Ncvick, M. Ro, & Grizzle, J. E. A Bayesian approach to the analysis of data from clinical trials. J. Amer. Stat. Ass., 1965, 60, 81-95. Pitz, G. F. An example of Bayesian hypothesis testing: the perception of rotary motion in depth. Psychol. Bull., 1968, 70, 252-255.

Ramsey, F. L. A Bayesian approach to bioassay. Biometrics, 1972, 28, 841-858. Restle, F. & Greeno, J. Introduction to Mathematical Psychology. AddisonWesley, Reading, Mass. 1970. Stael von Holstein, C-A. S. Assessment and evaluation of subjective probability distribution. The Economic Research Institute at the Stockholm School of Economics, Stockholm. 1970. Sutcliffe, J. P. A general method of analysis of frequency data for multiple classification designs. Psychol. Bull., 1957, 54, 154-137. Wald, A., Sequential Analysis. New York: Wiley, 1947. Weisberg, H. I. Bayesian comparison of two ordered multinominal populations. Biometrics, 1972, 28, 859-867. Willis, R, E. A Bayesian framework for the reporting of experimental results. Decision Sciences, 1972, 3 (4), 1-18. Winkler, R. L. The assessment of prior distributions in Bayesian analysis. J. Amer. Stat. Ass., 1967, 62, 776-800. Winkler, R. Introduction to Bayesian Inference and Decision. Holt-Reinhart & Winston, New York. 1972. 52

Distribution List Director, Engineering Psychology Programs Code 455 Office of Naval Research 800 North Quincy Street Arlington, Virginia 22217 Defense Documentation Center Cameron Station Alexandria,. Virginia 22314 (5 cys) (12 cys) Director, ONR Branch Office Attn: Dr. C. Harsh 495 Summer Street Boston, Massachusetts 02210 Director, ONR Branch Office Attn: Dr. M. Bertin 536 S. Clark Street Chicago, Illinois 60605 Director, ONR Branch Office Attn: Dr. E. Gloye 1030 East Green Street Pasadena, California 91106 Office of the Chief of Naval Operations, Op-095 Department of the Navy Washington, D. C. 20350 Dr. John J. Collins Office of the Chief of Naval Operations, Op-987F Department of the Navy Washington, D. C. 20350 CDR H. J. Connery Office of the Chief of Naval Operations, Op-987M4 Department of the Navy Washington, D. C. 20350 Dr. A. L. Slafkosky Scientific Advisor Commandant of the Marine Corps Code AX Washington, D. C. 20380 Mr. John Hill Naval Research Laboratory Code 5634 Washington, D. C. 20375 Office of Naval Research Mathematical Sciences Division Code 434 Department of the Navy Arlington, Virginia 22217 Director, Attn: Mr. 1030 East Pasadena, ONR Branch Office R. Lawson Green Street California 91106 Director, Naval Research Laboratory Technical Information Division Code 2027 Washington, D. C. 20375 (6 cys) Director, Naval Research Laboratory Attn: Library, Code 2029 (ONRL) Washington, D. C. 20375 (6 cys) Office of Naval Research Code 437 800 North Quincy Street Arlington, Virginia 22217 Office of Naval Research Code 463 800 North Quincy Street Arlington, Virginia 22217

Dr. Heber G. Moore Hqs., Naval Material Command Code 03R4 Department of the Navy Washington, D. C. 20360 Chief of Naval Material Prog. Admin. Personnel & Training NAVMAT 03424 Department of the Navy Washington, D. C. 20360 Commander, Naval Electronics Systems Command Command and Display Systems Branch Code 0544 Washington, D. C. 20360 Commander, Naval Air Systems Command NAVAIR 340F Washington, D. C. 20361 CDR James E. Goodson Bureau of Medicine and Surgery Operational Psychology Branch Code 513 Department of the Navy Washington, D. C. 20372 LCDR Curt Sandler, MSC Naval Safety Center Code 811 Norfolk, Virginia 23511 CDR Robert Wherry Human Factors Engineering Systems Office Naval Air Development Center Johnsville Warminster, Pennsylvania 18974 Dr. Gerald Miller Human Factors Branch Naval Electronics Laboratory Center San Diego, California 92152 Mr. James Jenkins Naval Ships Systems Command Code PMS 302-43 Washington, D. C. 20362 Naval Ships Systems Command Code 03H Washington, D. C. 20362 Commander, Naval Supply Systems Command Logistic Systems Research and Design Division Research and Development Branch Washington, D. C. 20376 Bureau of Medicine and Surgery Human Effectiveness Branch Code 713 Department of the Navy Washington, D. C. 20372 Mr. A. Sjoholm Bureau of Personnel Personnel Research Div., Washington, D. C. 20370 PERS A-3 Human Factors Engineering Branch Code 5342 Attn: LCDR R. Kennedy U. S. Naval Missile Center Point Mugu, California 93041 Human Engineering Branch, Code A624 Naval Ship Research and Development Center Annapolis Division Annapolis, Maryland 21402 Dr. Robert French Naval Undersea Center San Diego, California 92132 Mr. Richard Coburn Head, Human Factors Division Naval Electronics Laboratory Center San Diego, California 92152

Dean of Research Administration Naval Postgraduate School Monterey, California 93940 Mr. William Lane Human Factors Department Code N215 Naval Training Equipment Center Orlando, Florida 32813 U. S. Air Force Office of Scientific Research Life Sciences Directorate, NL 1400 Wilson Boulevard Arlington, Virginia 22209 Dr. J. M. Christensen Chief, Human Engineering Division Aerospace Medical Research Lab. Wright-Patterson AFB, Ohio 45433 Dr. Walter F. Grether Behavioral Science Laboratory Aerospace Medical Research Lab. Wright-Patterson AFB, Ohio 45433 Dr. J. E. Uhlaner Director U.S. Army Research Institute for the Social & Behavioral Sciences 1300 Wilson Boulevard Arlington, Virginia 22209 Dr. E. R. Dusek, Director Individual Training & Performance Research Laboratory U. S. Army Research Institute for the Behavioral & Social Sciences 1300 Wilson Boulevard Arlington, Virginia 22209 Dr. Jesse Orlansky Institute for Defense Analyses 400 Army-Navy Drive Arlington, Virginia 22202 Mr. Luigi Petrullo 2431 N. Edgewood Street Arlington, Virginia 22207 Commanding Officer (3 cys) Naval Personnel Research and Development Center Attn: Technical Director San Diego, California 92152 Dr. George Moeller Head, Human Factors Engineering Branch Submarine Medical Research Lab. Naval Submarine Base Groton, Connecticut 06340 Lt. Col. Austin W. Kibler Director, Behavioral Sciences Advanced Research Projects Agency 1400 Wilson Boulevard Arlington, Virginia 22209 Chief of Research and Development Human Factors Branch Behavioral Science Division Department of the Army Washington, D. C. 20310 Attn: Mr. J. Barber Dr. Joseph Zeidner, Director Organization & Systems Research Lab. U. S. Army Research Institute for the Behavioral & Social Sciences 1300 Wilson Boulevard Arlington, Virginia 22209 Technical Director U. S. Army Human Engineering Laboratories Aberdeen Proving Ground Aberdeen, Maryland 21005 Dr. Stanley Deutsch Chief, Man-Systems Integration OART, Hqs., NASA 600 Independence Avenue Washington, D. C. 20546 Capt. Jack A. Thorpe Department of Psychology Bowling Green State University Bowling Green, Ohio 43403

Dr. Eugene Galanter Columbia University Department of Psychology New York, New York 10027 Dr. J. Halpern Department of Psychology University of Denver University Park Denver, Colorado 80210 Dr. S. N. Roscoe University of Illinois Institute of Aviation Savoy, Illinois 61874 Dr. William Bevan The Johns Hopkins University Department of Psychology Charles & 34th Street Baltimore, Maryland 21218 Dr. James Parker Bio Technology, Inc. 3027 Rosemary Lane Falls Church, Virginia 22042 Dr. W. H. Teichner Department of Psychology New Mexico State University Las Cruces, New Mexico 88001 Dr. Edwin A. Fleishman American Institutes for Research 8555 Sixteenth Street Silver Spring, Marylan 20910 American Institues for Research Library 135 N. Bellefield Avenue Pittsburgh, Pa. 15213 Dr. Joseph Wulfeck Dunlap and Associates, Inc. 1454 Cloverfield Boulevard Santa Monica, California 90404 Dr. L. J. Fogel Decision Science, Inc. 4508 Mission Bay Drive San Diego, California 92112 Psychological Abstracts American Psychological Association 1200 17th Street Washington, D. C. 20036 Dr. Irwin Pollack University of Michigan Mental Health Research Institute 205 N. Forest Avenue Ann Arbor, Michigan, 48104 Dr. W. S. Vaughan Oceanautics, Inc. 3308 Dodge Park Road Landover, Maryland 20785 Dr. D. B. Jones Martin Marietta Corp. Orlando Division Orlando, Florida 32805 Mr. Wes Woodson Man Factors, Inc. 4433 Convoy Street, Suite D San Diego, California 92111 Dr. Robert R. Mackie Human Factors Research Inc. Santa Barbara Research Park 6780 Cortona Drive Goleta, California 93017 Dr. A. I. Siegel Applied Psychological Services 404 East Lancaster Street Wayne, Pennsylvania 19087 Dr. Ronald A. Howard Stanford University Stanford, California 94305

Dr. Amos Freedy Perceptronics, Inc. 17100 Ventura Boulevard Encinco, California 91316 Dr. Paul Slovic Department of Psychology Hebrew University Jerusalem, Israel Dr. C. H. Baker Director, Human Factors Wing Defense & Civil Institute of Environmental Medicine P. O. Box 2000 Downsville, Toronto, Ontario Canada Dr. D. E. Broadbent Director, Applied Psychology Unit Medical Research Council 15 Chaucer Road Cambridge, CB2 2EF England Dr. Cameron R. Peterson Decision and Designs, Inc. Suite 600 7900 Westpark Drive McLean, Virginia 22101 Dr. Victor Fields Montgomery College Department of Psychology Rockville, Maryland 20850 Dr. Robert B. Sleight Century Research Corporation 4113 Lee Highway Arlington, Virginia 22207 Journal Supplement Abstract Service American Psychological Association 1200 17th Street, N. W. Washington, D. C. 20036 Dr. Bruce M. Ross Department of Psychology Catholic University Washington, D. C. 20017 Dr. David Meister U. S. Army Research Institute 1300 Wilson Boulevard Arlington, Virginia 22209 Mr. John Dennis ONR Resident Representative University of Michigan Ann Arbor, Michigan Dr. Howard Egeth Department of Psychology The Johns Hopkins University 34th & Charles Streets Baltimore, Maryland 21218 Capt. T. A. Francis Office of the Chief of Naval Operation, Op-965 Room 828, BCT #2 801 North Randolph Street Arlington, Virginia 22203 Dr. Sarah Lichtenstein Department of Psychology Brunel University Kingson Lane Uxbridge, Middlesex England

UN IVER 1Y OF MICHN 3 1 J t