ENGINEERING RESEARCH INSTITUTE UNIVERSITY OF MICHIGAN ANN ARBOR 1TE EVIDENCE FOR A DECISION-MAKING THEORY OF VISUAL DETECTION Technical Report No. 40 Electronic Defense Group Department of Electrical Engineering By: J. A. Swets Approved by: W. P. Tanner, Jr. A. B. Macnee T. G. Birdsall Project 2262 TASK ORDER NO. EDG-3 CONTRACT NO. DA-36-039 sc-63203 SIGNAL CORPS, DEPARTMENT OF THE ARMY DEPARTMENT OF ARMY PROJECT NO. 3-99-04-042 SIGNAL CORPS PROJECT NO. 194B February, 1955

The experiments reported in this paper were sponsored jointly by the Electronic Defense Group under Contract No. DA-36-039 sc63203 and the Vision Research Laboratory under Contract No. DA-36-039 sc52654 and Contract No. Nobs 53242. Therefore, this report appears also as a portion of Report 2144-20-T from the Engineering Research Institute, a report from the Optics and Vision Group of Project Michigan. Report 2144-20-T differs from the present report principally in that the relevance of the research to a classified military problem is indicated. ii

ACKNOWIEDGEMENT The authors wish to acknowledge the valuable contributions of some of our friends and associates without whose assistance this work would have been impossible. Dr. H. R. Blackwell, director of the Vision Research Laboratory, contributed freely of his time in discussions during every stage of the work. In permitting the use of his laboratory, he made possible the successful completion of the experiments. Dr. W. M. Kincaid, of the staff of the Vision Research Laboratory, exhibited a lively interest throughout the progress of this work. He was always available for consultation, and contributed many valuable suggestions. The authors take this opportunity to express their gratitude to Dr. Blackwell and Dr. Kincaid for their assistance. iii

TABLE OF CONTENTS Page LIST OF ILLUSTRATIONS v LIST OF TABLES vi ABSTRACT vii 1. INTRODUCTION 1 1.1 Related Articles 1 1.2 A Decision-Making Theory of Visual Detection 2 1.2.1 The Fundamental Problem 3 1.2.2 Assumptions Made in Applying the Theory of Signal Detectability to the Behavior of the Human Observer 3 1.2.3 The Definition of Criterion and Likelihood Ratio 4 1.2.4 The Essence of the Theory of Signal Detectability 5 1.3 Scope of Discussion 6 1.4 The Incompatibility of a Decision-Making Theory of Visual Detection and Conventional Sensory Theory 8 2. THE EXPECTED-VALUE OBSERVER 10 2.1 Data from the First Experiment 13 2.2 Data from the Second Experiment 21 2.3 The Approach to the Optimum Behavior 29 3. THE NEYMAN-PEARSON OBSERVER 31 3.1 The Approach to the Announced Optimum Value of P (A) 31 3.2 Orher Analyses of the Data of this Study 32 4. THE A POSTERIORI PROBABILITY OBSERVER 34 4.1 The Relationship between d(x) and l(x) 36 5. THE SECOND-CHOICE EXPERIMENT 43 5.1 The Rationale for the Second-Choice Experiment 44 5.2 Results 45 5.3 A Note on the Variance Assumption 50 6. DISCUSSION 51 6.1 The Possibility of a Threshold Theory 53 6.2 Some Implications of the Proposed Model for Psychological Theory 54 6.3 Some Implications for Practice 55 iv

LIST OF ILLUSTRATIONS Page Fig. 1 PSN(A) vs. PN(A) with d' as the Parameter 12 Fig. 2 PSN(A) vs. PN(A) for Observer 1 in the First Expected-Value Experiment 14 Fig. 3 PN(A) vs. PN(A) for Observer 2 in the First Expected-Value Experiment 15 Fig. 4 PSN(A) vs. PN(A) for Observer 3 in the First Expected-Value Experiment 16 Fig. 5 Raw Data, Average for all Observers in the First Expected-Value Experiment 22 Fig. 6 Raw Data Corrected for Chance, Average for All Observers in the First Expected-Value Experiment 23 Fig. 7 PSN(A) vs. PN(A) for Observer 1 in the Second Expected-Value Experiment 24 Fig. 8 PSN(A) vs. PN(A) for Observer 2 in the Second Expected-Value Experiment 25 Fig. 9 PSN(A) vs. PN(A) for Observer 3 in the Second Expected-Value Experiment 26 Fig. 10 PSN(A) vs. PN(A) for Observer 4 in the Second Expected-Value Experiment 27 Fig. 11 Objective Probability vs. Subjective Probability for Observer 1 38 Fig. 12 Objective Probability vs. Subjective Probability for Observer 2 39 Fig. 13 Objective Probability vs. Subjective Probability for Observer 3 40 Fig. 14 Objective Probability vs. Subjective Probability for Observer 4 41 Fig. 15 The Second-Choice Data and Predictions for the Second-Choice Situation Under the Assumption that the N and S + N Distributions Are Gaussian and of Equal Variance 46 v

Fig. 16 The Second-Choice Data and Predictions for the Second-Choice Situation Under the Assumption that AM 49 = 4 Fig. 17 Block Diagram of the Expected-Value Observer 51 LIST OF TABLES Table 1 Yes-No Data for Observer 1 in the First ExpectedValue Experiment 18 Table 2 Yes-No Data for Observer 2 in the First ExpectedValue Experiment 19 Table 3 Yes-No Data for Observer 3 in the First ExpectedValue Experiment 20 Table 4 Data from 4 Observers in Second Expected-Value Experiment 28 Table 5 Neyman-Pearson Observer 33 Table 6 A Posteriori Probability Observer 35 Table 7 The Relationship between d(x) and l(x) 42 vi

ABSTRACT This is one of a series of papers concerned with the psychophysical application of the mathematical theory of signal detectability. This paper brings together all of the data on visual detection collected to date that bear directly on the case of the signal-known-exactly as treated by the theory of signal detectability. The general conclusion drawn from the experimental results reported is that the model provided by the theory of signal detectability or, more generally, by the theory of statistical decision, is applicable to the visual detection behavior of the human observer. That is to say, the human observer is capable of ordering values of the variable upon which detection depends well below the threshold level as this level is conventionally conceived. Further, the experiments show the observer to be capable of behaving in accordance with several kinds of optimum decision as defined within the theory of signal detectability. The general implications of the applicability of the model for sensory theory and psychophysical methods are discussed. vii

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN THE EVIDENCE FOR A DECISION-MAKING THEORY OF VISUAL DETECTION 1. INTRODUCTION This is one of a series of papers concerned with the psychophysical pplication of the mathematical theory of signal detectability. This introductory action contains a short discussion 6f the references in which the theory of sig31 detectability is presented, and also contains references to other papers in he series on the psychophysical application of the theory of signal detectability. dere follows, in this introductory section, a brief description of the theory of ignal detectability and of the theory of visual detection based on it, and an utline of the scope of discussion of the present paper..1 Related Articles "The Theory of Signal Detectability," by Peterson and Birdsall, appears s Technical Report No. 13 of the Electronic Defense Group of the University of ichigan (Ref. 15). Technical Report No. 19, by Fox (Ref. 8), contains a nonathematical discussion of the material of Technical Report No. 13 on the general heory of detectability, as well as a discussion of the methods employed in seuential observation processes. Birdsall's Technical Report No. 20 (Ref. 2) preents an application of game theory to signal detectability. The material of eports 13 and 19 is also available in the "Transactions of the I.R.E. Profesional Group on Information Theory", (Ref. 16). The application of the theory of signal detectability to visual detecion by the human observer was first reported by Tanner and Swets in Technical eport No. 18 of the Electronic Defense Group (Ref. 22). This topic is also disussed elsewhere (Refs. 19, 25, 24). The most readily available source containing

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN a report of some of this work is the Psychological Review of November, 1954 (Rej 25). A paper in the "Transactions of the I.R.E. Professional Group on Informat: Theory" (Ref. 21) contains a report of the work of Tanner and Norman relating ti theory of signal detectability to auditory detection. Several other papers in this series are currently being prepared. Tec nical Report No. 30 by Tanner, Swets, and Green (Ref. 26) is another paper relal ing the theory to data from auditory experiments. Technical Report No. 42 by Tanner, Birdsall, and Swets (Ref. 20) deals with the implications of this research for methods and analysis procedures in psychophysical testing. The present paper brings together the data on visual detection collect to date that bear directly on the case of the signal-known-exactly as treated bJ the theory of signal detectability. Some of these data have been reported previously only in relatively inaccessible sources (Refs. 19, 22, 23, and 24); some of the data that were reported in a more available source (Ref. 25) were only briefly described there. In addition, some of the data presented below have nol been previously reported in any form. It is the primary purpose of this paper t present all relevant available data; although the theory of signal detectability and the theory of visual detection based on it are reviewed, they are not discussed completely, and whereas the general implications for sensory theory, methods, and data analysis are drawn, they are not treated here in full detail. 1.2 A Decision-Making Theory of Visual Detection In this section, the theory of signal detectability is reviewed as it relates to the problem of visual detection by the human observer. It should be pointed out, by way of introduction, that the theory of signal detectability is derived directly from the theory of statistical decision, the theory of testing statistical hypotheses. This latter theory is presented most completely by Wald 2

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN ef. 28); important papers published earlier include those by Wald (Ref. 27), yman (Ref. 13), and Neyman and Pearson (Ref. 14). 1.2.1 The Fundamental Problem. The fundamental problem in signal detecbility involves a fixed observation interval; the observer is presented a eceiver input", a function of time for T seconds. He is then asked to decide, the basis of the observation, whether that receiver input arose from noise one, or from signal plus noise, where the signal is known to be from a certain semble of signals. This problem has an exact parallel in visual psychophysics. The obserr in the most common visual psychophysical experiment, an experiment employing e so-called "yes-no" method, is asked to observe, at a particular time and a rticular location in the visual field, a signal of a particular size, duration, d intensity. Ordinarily the size and duration of the signal, as well as its cation and time of occurrence, are known by the observer; in certain instances, formation about the intensity of the signal and the a priori probability of gnal occurrence are also provided the observer. He is then asked to state, on.e basis of the observation, whether or not a signal was presented. 1.2.2 Assumptions Made in Applying the Theory of Signal Detectability the Behavior of the Human Observer. Given certain assumptions, asking the obrver in a psychophysical experiment to state whether or not a signal was prented is equivalent to asking him to decide whether his observation arose from gnal plus noise or from noise alone, or stated another way, to decide whether or t to accept the hypothesis that a signal existed. The primary assumption is that.e value of the variable upon which detection depends, presumably neural activity, ries from instant to instant in a random fashion when no signal is present, and at the value of this variable produced by a signal of given strength is also 3

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN randomly distributed. Although there is no need to conceptualize this variable in neurophysiological terms, it may be helpful, and the assumption stated is su gested by present knowledge of neurophysiology. So it may be maintained on a priori grounds, that if the problem of visual detection is the detection of signals having randomly distributed neural effects, in the presence of a background of random interference, then the theory of statistical decision, or in p, ticular, the theory of signal detectability, constitutes a model of possible re. vance to visual detection. An attempt to apply the theory of signal detectability to human behavior, therefore, implies the assumptions that the sensory systems function pr marily to transmit information, and that the sensory systems are noisy channels An additional assumption, made apparent in the following pages, is that the cen tral mechanisms involved in decision making are capable of making optimal use o: the information transmitted by sensory paths. It is assumed, then, that the observer operates with a continuous variable, the values of which constitute "observations," and that any value of this variable may arise either from noise alone or from signal plus noise. It assumed that, when the signal ensemble is known to the observer, the probabilit: that a given observation represents noise alone, and the probability that this value arose from the signal-plus-noise distribution, can be estimated by him. Thus, it is assumed that the observer in a "yes-no" experiment must establish a level of confidence, or criterion, and base his decision on the rel~ tion of the observation to this criterion. 1.2.3 The Definition of Criterion and Likelihood Ratio. According t( the theory, the observer chooses a set of observations (the criterion A) such tl an observation in this set will lead him to Accept the existence of a signal, tl' —-------------- 4

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN the observer accepts the hypothesis that signal plus noise existed during the servation interval. All other observations are in the Complement of the cririon, CA; these are regarded by the observer as representing noise alone. SN 11 be used to denote signal plus noise and N will denote noise alone. If there e only a countable number of possible observations, each observation, x, having,obability PSN(x) of occurrence if there is signal plus noise present, and probility PN(x) of occurrence if noise alone is present, then the likelihood ratio defined as!(x) = PsN(x)/PN(x). Here, x will be considered continuous, and'obability density functions (frequency functions) fsN(x) and fN(x) are used; cordingly l(x) = fsN(x)/fN(x). It is assumed that for every x the observer can timate l(x) which is the relative likelihood that x arose from signal plus noise compared to the possibility that x arose from noise alone. A criterion may be evaluated in terms of the integrals of the density inctions over the criterion A, since the integral of fsN(x) over A is the con-.tional probability of detection, PSN(A), and the integral of fN(x) over A is the )nditional probability of a false alarm (a Type I error in statistical parlance), I(A). 1.2.4 The Essence of the Theory of Signal Detectability. The essence the theory of signal detectability is the definition of a class of criteria in rrms of likelihood ratio. Under each of several definitions of the optimum crinrion, the optimum is found to be in this class of likelihood-ratio criteria. A ~iterion in this class is denoted A(P); that is, the criterion A contains all )servations with likelihood ratio greater than or equal to 3, and none of those th likelihood ratio less than 3. The solution, then, with respect to a given ~finition of optimum, is the exact value of P to be used. f is defined as the )erating level of the likelihood-ratio receiver. 5

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN It should be noted that the theory of signal detectability specifies as the optimum receiver that receiver whose output is either likelihood ratio or some monotonic function of likelihood ratio. If the output of the receiver is likelihood ratio, then the solution, for each definition of optimum, is the criterion with the proper operating level P. If the receiver's output is some decision function other than likelihood ratio, but a monotonic function of likelihood ratio, then the optimum operating level is the value of the monotonic function at P. That is, if the receiver output is some function d(x) = F [ (x)], where F is strictly monotonic, then the optimum criterion is specified by i' = F(P), such that d(x)> l'- — (x)> B. Thus, the theory of signal detectability may describe the behavior of the human observer if the human observer operates with a continuous variable, or decision function, that is either likeli. hood ratio or some monotonic function of likelihood ratio. 1.3 Scope of Discussion Peterson, Birdsall, and Fox (Ref. 16) advance six definitions of optimi and their solutions. Experiments have been performed, and are reported in the next three sections of this paper, to test the ability of the human observer to act in accordance with three of these definitions of optimum. These three definitions of optimum and their respective solutions are listed here. 1. Expected-Value Criterion - that criterion that maximizes the tota: expected value, where the individual values are: VSNA = the value of a detection SN.A V = the value of a correct rejection N*CA K = the cost of a miss SN CA CN.A = the cost of a false alarm ------------------------ 6 —--------

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN P(N) VN.CA + KNA lution: A() = P(SN) VSN.A + KSN.CA ere P(N) and P(SN) are the a priori probabilities. 2. The Neyman-Pearson Criterion - that criterion such that PSN(A) is maximum, while PN(A)k. lution: A( ) where PN[A( k)] = k 3. A Posteriori Probability - not a criterion but the best estimate of e probability that the observation arose from signal plus noise. (x) P(SN) Px(SN) =.....e(x) P(SN) + P(N) her definitions of optimum, not considered explicitly here, for which the soluon has been provided (Ref. 16) include the Weighted-Combination criterion, the iterion that maximizes PSN(A) - w PN(A); Siegert's Ideal Criterion, the criterior at minimizes total error; and the Information Criterion, the criterion that ximizes the reduction in uncertainty, in the Shannon sense (Ref. 18), as to ether or not a signal was sent. It should be pointed out, however, that the ighted-Combination Criterion is the abstract criterion, of which the Expectedlue Criterion and Ideal Criterion are special cases. The Expected-Value Cririon is identical to the Weighted-Combination Criterion for the case where = D and is identical to the Ideal Criterion for the case where VN.CA + KNA VsN-A + KSNCA 7

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN The fifth section of this paper contains the results of an experiment designed to determine the congruence of the behavior of the human observer and the definition of optimum behavior for the forced-choice situation, a definition of optimum that is not treated explicitly in the theory of signal detectability. 1.4 The Incompatibility of a Decision-Making Theory of Visual Detection and Con ventional Sensory Theory Anticipating to an extent that will facilitate subsequent description: the primary result of all the experiments performed is that the theory of signal detectability, or more generally, the theory of statistical decision, is applica ble to the behavior of the human observer. That is to say, the experiments demonstrate that the human observer operates with a decision function that is either likelihood ratio or some monotonic function of likelihood ratio, and that the human observer tends to behave optimally. This result implies, of course, the ability to discriminate among obse vations, or values of the decision function, that may result from noise alone Thus, if a threshold (fixed operating level) exists, this threshold is low enoug to be exceeded by noise alone an appreciable portion of the time. The concept o a threshold that is exceeded more than very rarely by noise alone is quite diffe ent from the concept of threshold that is an integral part of conventional senso theory. The primary purpose of this paper is the presentation of data; hence, this is not the place to attempt a detailed discussion of the notion of a thresh old as it appears in sensory theory. It may be well, however, to adduce one or two instances of current methods in sensory experimentation in order to support the statement that the threshold as conceived in sensory theory is a relatively fixed level that is rarely, if ever, exceeded by noise alone. 8

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN Many studies of sensory processes do not employ trials in which no sigil is presented. In these cases, it is clearly not regarded as important to 3sess the probability that the fixed level be exceeded by noise alone. In other budies, the experimenter may occasionally insert one or two trials in which no Lgnal is presented (or occasionally turn off a continuous signal) in order to 3tect what are regarded as spurious responses, so that he may caution the obser-;rs against such responses. Such trials have been referred to as vexirfehlen, term which may be reasonably translated as "catch signals". A refinement of the rocedure in which catch signals are sporadically presented, one that is used 3ther frequently, is to fail to present signals on a larger proportion of trials i order to assess quite accurately the extent of "yes" responses on such trials. ien, however, not only is this information subtracted from the data, but the ctent of "yes" response to catch signals is used in estimating the amount of 3purious" responses made to actual signals so that these, too, may be eliminated rom the data. This use of the correction for chance in psychophysical experimnts has been described in Ref. 25. Regarding the totality of "yes" responses D catch signals as spurious, and regarding such spurious responses to be equally Lkely for all values of signal intensity, is valid only if the threshold level; such that it is exceeded by noise alone on a negligible proportion of the rials. The validity of the application of the chance correction to psychophysiil data depends upon the validity of the assumption that "yes" responses to catch Lgnals are spurious responses, or random guesses, and that these responses are Idependent of "sensory-determinate" responses. The theory advanced here, on the:her hand, assumes a dependence between the conditional probability that an I. R. Blackwell, unpublished manuscript. 9

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN observation arising from signal plus noise will be in the criterion and the conditional probability that an observation arising from noise alone will be in the criterion. The data presented in this paper, to the extent that they are analyzed in this paper, do not indicate how far down into the noise the observers actuall ordered their observations, that is, how low relative to the distribution of observations arising from noise alone a fixed threshold must be to be compatible with the data. It may be the case, then, that further analysis will show the present data or new data to be consistent with a fixed threshold, say, at the me of the noise distribution, a threshold exceeded by noise alone approximately 50 percent of the time. It can be stated, on the basis of completed analyses of th present data which are to be reported in a paper which considers and evaluates several alternative models (Ref. 20), that a fixed threshold, if one exists, mus be lower than plus one sigma from the mean of the noise distribution, that is, must be exceeded by noise alone more than 16 percent of the time. It is importa that the reader note that, unless specifically stated otherwise, further references in this paper to the threshold refer to a threshold in the conventional sense, that is, to a threshold exceeded, to be conservative, less than 5 percent of the time by noise alone. The experiments were designed to detect a threshold at approximately this level if one existed. 2. THE EXPECTED-VALUE OBSERVER An experiment concerning the ability of the human observer to maximize the total expected value is the one experiment discussed elsewhere in a readily accessible source (Ref. 25). This section supplements the discussion in Referer 25 in that it presents in more detail the data that were necessarily described only briefly there. In addition, this section contains the previously unpublisi 10

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN =sults of further experimentation on the Expected-Value Observer that is superior a certain respects. The analysis of data, in terms of the theory of signal detectability, 3kes the form of plots of what are called ROC curves _ Receiver Operating aaracteristics. An ROC curve is a plot of the conditional probability of detecion, PSN(A), against the conditional probability of false alarm, PN(A). A comLete curve is obtained if all possible values of operating level, or P, are con[dered. Peterson and Birdsall (Ref. 15) have demonstrated that the optimum )erating level is represented by a point on the ROC curve where its slope is 3. The typical ROC curve, one that has occurred frequently in this work, s shown in Fig. 1. Frequently, as in this figure, a family of ROC curves is lotted with signal strength as the parameter. In this particular case, the arameter is d', the index used for the analysis of data in terms of the theory f signal detectability.1 It is defined as the difference between the means of he noise and signal-plus-noise distributions normalized to the standard deviation f the noise distribution that is, d' = MSN - MN Thus, d' can be conceived ~~x a~~~4T f as a standard score, an - measure, or again, it may be thought of as a output) signal-to-noise ratio. A more complete description of d' and of the ROC urves may be found elsewhere (Refs. 15, 16, 20). If a threshold exists, this fact is immediately apparent from the ROC aives. The conventional notion of a fixed criterion or threshold, in the termiDlogy of this paper, implies the existence of a set of observations leading d' is essentially a dependent variable; the paper on psychophysical methods (Ref. 20) treats explicity the reasons for preferring it to the usual dependent variable in psychophysical experiments; namely, the calculated threshold or minimum detectable signal. The primary purpose of the present paper is to present the data upon which the recommendations made in the paper on psychophysical methods are based. 11

r9-I-1t-, W3P r31-~9-V OZ61 1.0 0.9 0. 8 0.7 0.6 0.5 U), Z 0.4 0.3 0,2 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 I. C PN (A) FIG. I. FN(A) VS. F(A) WITH d' AS THE PARAMETER

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN an extremely small probability of false alarm. The observer's criterion inLudes this set and may or may not include a random selection of other observation iat is, "guesses". The ROC curve in this case, as pointed out elsewhere (22, 25),; a straight line from the left-hand vertical axis to the upper right-hand corner..1 Data from the First Experiment The ROC curves obtained from the first experiment on the Expected-Value bserver, the experiment discussed in reference 25, are displayed here in Figures, 3, and 4. Five values of signal intensity, including the zero-intensity sig1l or "blank", were used in this experiment. It may be seen from an examination f these plots that the data of the first experiment do not provide an adequate 2finition of the entire curve. As a matter of fact, even in the region where lints have been obtained (roughly from PN(A) = 0.0 to PN(A) =.50), the curve is st well defined. This latter inadequacy may be attributed, in part, to the small umber of observations per point. A second factor which operated to cause disarsion of these points was the day-to-day variation in signal and background atensities which, unlike in the calculation of contrast thresholds, is not taken ato account in the present analysis. The effects of both of these sources of ariance were reduced, to a large extent, in the second Expected-Value experiment iich is reported below. Although entire ROC curves were not precisely defined by the data from ae first experiment, these data are entirely adequate for the purpose of disinguishing between the predictions of a theory based on the model provided by ie theory of signal detectability and the predictions which follow from the con~pt of a threshold or fixed criterion, the purpose for which they were used in ~ference 25. It is clear, for example, that the straight lines fitted to the 13

4 3 2.9 - Xi 8 o x x X X7 -.6 1.2.3.5 6 7 9 1.0 0 0.2.2 3 4.5 6 7.8 9.(A) PN(A) FIG. 2. PSN(A) VS. PN(A) FOR OBSERVER I IN THE FIRST EXPECTED-VALUE EXPERIMENT. 14

__ Q _3 04 2 3 1.0.9 -- _/X.8 / I X X.7 c.5 —.4.3.2 j, ) CD.1 n o I 0.1.2.3.4.5.6.7.8.9 1.0 PN (A) FIG. 3. PSN(A) VS. PN(A) FOR OBSERVER 2 IN THE FIRST EXPECTED-VALUE EXPERIMENT. 15

3 42 1.0 x x 0x/ x x.8 /APNA) x 7,6.4t X 0.2 2 0 — 0.1.2.3.4.5.6.7.8.9 1.0 PN A) FIG. 4. PSN(A) VS. PN(A) FOR OBSERVER 3 IN THE FIRST EXPECTED-VALUE EXPERIMENT. 16

- ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN.ata in Figures 2, 3, and 4 do not intersect the upper right-hand corner of the;raph, as required by the concept of a fixed threshold. These straight lines Lppear, rather, to be arcs of the type of ROC curve predicted by the theory of:ignal detectability. The data displayed graphically in Figures 2, 3, and 4 are'eported more precisely in Tables 1, 2, and 3. On days 1 through 8, there were 50 presentations per day of each value )f signal intensity including the zero-intensity signal. On days 9 through 12,,here were 60 presentations per day at each intensity. On days 13 through 16,;here were 30 presentations per day of each value of signal intensity greater;han zero, and 180 blanks per day. This experiment, like the other experiments'eported in this paper, employed a circular target, 30 minutes of visual angle.n diameter, with a duration of 1/100 second, on a ten foot-lambert background.'he more general aspects of the procedure and apparatus involved in this experiient and the other experiments reported in this paper are discussed in Reference?5 and in an article by Blackwell, Pritchard, and Ohmart (Ref. 4). Tables 1, 2, and 3 also contain the data which serve as a basis for the Coefficients of correlation that are reported in References 22 and 25 and again )elow, between PN(A) and calculated threshold. The implication of these correlations is the same as the implication of the straight lines fitted to the data of figures 2, 3, and 4, namely, that a dependence exists between the conditional probability that an observation arising from SN will be in the criterion and the conditional probability that an observation arising from N will be in the criterion - a condition that, as described elsewhere (Refs. 22, 24, 25), is inconsistent with the concept of threshold, or fixed criterion. The product-moment correlations for the three observers between PN(A) nd calculated threshold, based on the twelve sessions involving a payoff matrix ---- 17

DAY P(SN) VSN A KSNCA KN A VCA PN(A) PSCN(A) P2N(A) PSN(A) PS(A) Chres * N.CA 1 2 S.~3 S1N Threshold 1.8 --- --- -- 0.50 0.62 0.76 0.90 0.94.68 2.8 ---.44.46.74.84.94.73 3.4 --- --- -.18.40.64.76 1.00.68 4.4 --- -- -- --.29.32.48.80.96.81 5.8 +1 -1 -2 +2.44.50.54.86.98.74 6.8 +1 -1 -2 +2.12.26.32.68.98.73 7.8 +1 -1 -2 +2.36.42.58.72.96.71 8.8 +1 -1 -2 +2.38.44.50.74.84.73 cc 9.8 +1 -1 -2 +2.30.42.53.82 1.00.78 10.8 +1 -1 -2 +2.20.37.53.75.97.75 11.8 +1 -1 -3 +3.17.27.47.73.98.78 12.8 +1 -1 -4 +4.07.08.17.53.89.91 13.4 +1 -1 -2 +2.05.07.30.63.97.86 14.4 +1 -1 -1 +1.04.17.43.80.97.74 15.4 +2 -2 -1 +1.43.47.63.90 1.00.76 16.4 +3 -3 -1 +1.54.60.80.80 1.00.80 TABLE 1

DAY P(SN) VSN.A KSN.CA KNA VNCA P(A) PS (A) P2N(A) PS (A) P (A) Contrast N.A N S1NN S3N S4N Threshold 1.8 -- - -.38.56.76.86.98.62 2.8 - - -.46.72.64.88.98.66 3.4 - - - -.17.24.56.64 1.00.76 4.4 -- -- -.15.12.60.72.96.77 5.8 +1 -1 -2 +2.36.54.60.92.96.63 6.8 +1 -1 -2 +2.18.20.40.76 1.00.77 7.8 +1 -1 -2 +2.24.34.56.76.92.66 F8.8 +1 -1 -2 +2.42..50.66.68.90.68 9.8 +1 -1 -2 +2.23.23.47.72.95.84 10.8 +1 -1 -2 +2.37.43.63.85.95.76 11.8 +1 -1 -3 +3.42.47.65.93 1.00.74 12.8 +1 -1 -4 +4.25.20.42.77.97.81 13.4 +1 -1 -2 +2.04 - -.33.80 1.07 14.4 +1 -1 -1 +1.03.07.27.60.97.85 15.4 +2 -2 -1 +1.26.20.57.80.97.77 16.4 +3 -3 -1 +1.35.40.50.87.97.84 TABLE 2 YES-NO DATA FOR OBSERVER 2 IN THE FIRST EXPECTED-VALUE EXPERIMENT

DAY j P(SN) VSN-A KSN CA K NA KN.CA PN(A) PS1N(A) PS2N(A) PS3N(A) C PS4No(A)st 1.8 --- --- --- --.42.54.66.86 1.00.72 2.8 --- -- --- ---.40 30.62.90 1.00.74 3.4 --- --- - ---.08.08.28.60.96.84 4.4 --- --- --- --.03.24.32.48.96.78 5.8 +1 -1 -2 +2.20.44.64.84.96.61 6.8 +1 -1 -2 +2.18.34.38.84.96.70 7.8 +1 -1 -2 +2.08.22.48.74.98.63 8.8 +1 -1 -2 +2.18.32.46.74.94.63 9.8 +1 -1 -2 +2.17.15.47.85 1.00.76 10.8 +1 -1 -2 +2.15.35.55.90.9766 11.8 +1 -1 -3 +3.07.10.35.75.9781 12.8 +1 -1 -4 +4 ---.07.22.62.95.83 13.4 +1 -1 -2 +2 ---.13.17.42.93.89 14.4 +1 -1 -1 +1 -- ---..20.37.80 1.00 15.4 +2 -2 -1 +1.24.47.77.90.9754 16.4 +3 -3 -1 +1.29.37.77.83.97 ~67 TABLE 3 YES-NO DATA FOR OBSERVER 3 IN THE

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN )ays 5-16), are -.37(p =.245), -.60(p =.039) and -.81(p =.001). For the three ibjects combined, p =.0008. The dependence of false-alarm rate and calculated threshold is also ilistrated, though somewhat crudely, in Figures 5 and 6. These plots represent the ita of days 5-16 for all three observers. The portion of data comprising each of ie curves was selected to be relatively homogeneous with respect to PN(A). Figure shows the raw data; the proportion of positive responses is plotted as a function ~AI, the signal intensity. Figure 6 shows the same data after application of the lance correction; here the calculated threshold (the value of AI corresponding to ie corrected proportion of.50) is seen to be dependent upon PN(A) in the preLcted direction. The observer, apparently, can adjust his criterion. If he )erates with a lower value of 8, the proportion of correct detections will be icreased by an amount that is not completely eliminated by the coincident increase i the size of the chance correction factor, PN(A). 2 Data from the Second Experiment Since the data from the first experiment did not suffice to trace out a )mplete ROC curve, a second experiment was conducted to obtain a broader range of Llues of PN(A). A different set of observers was used in this experiment, and ily one value of signal intensity was employed. The data for the four observers -e shown in Figures 7 through 10. Each point represents a two-hour observing 3ssion including 200 presentations of signal and 200 observations in which no gnal was presented; that is, P(SN) was held at.50 throughout this second experi-!nt. Changes in B, and thus in PN(A), were effected entirely by changes in the blative values and costs with which an Expected-Value Observer operates. The data ~om which Figures 7 through 10 were plotted are presented in Table 4. The column 21

9S-:-I WI3r SZI-~9-V Z9ZZ I.0 0.9 0 DAYS 15 -16 0.7 / w cn / > 0.5' 0.4 -- U- 04 0 22

GS-~-I W3r 9L 1-s9-V Z9ZZ 1.0 0.9 - DAYS 5-11 wl 0 DAYS 12-14 z 0.8 0 DAYS 15-16 ~ 0.7 0a~I 0,NST ( w ~ 0.6 C. / SIGNAL I T: PN(A:0.25 EXPECTED-VALUE EX PERI 0.04EN. 23 Mo — y ~/ O 0/ 0.2 0. 0 0.1 0.2 03 0.4 0.5 0.6 0- 0.8 0.9 1.0 SIGNAL INTENSITY (At) FIG. 6. RAW DATA CORRECTED FOR CHANCE, AVERAGE FOR ALL OBSERVERS IN THE FIRST EXPECTED-VALUE EXPERIMENT. 23

0.9 o 2' 0.7 0.S-6-6 3P Z1-_9-V Z9Z n 0.5 z a. 0.4 GQ3 - 1 71L OBSERVER I PN(A) VS. PSN(A) ASSUMING C 4 0.2 M — M WHERE d'= / N 0 0.1 0.2 03 0.4 Q5 0.6 0.7 Q8 0.9 l0 PN(A) FIG. 7. PSN(A) VS. PN(A) FOR OBSERVER I IN THE SECOND EXPECTED-VALUE EXPERIMENT. 24

t9-9-6 IJ3P 61-~29-V 9Szz 0.9 I -- O, —0.84 7E OBSERVER 2 WHERE d'. SN- N N_ 0.1 0 0.1 0.2 0.3 0.4 0.5 Os 0.7 0.8 0.9 1.0 PN(A) FIG. 8. PSFA) VS. PN(A) FOR OBSERVER 2 IN THE SECOND EXPECTED-VALUE EXPERIMENT. 25

bS-8-6 W3P Ogi-~9-V 29zz 1.0 0.9 1.0 0.83 BSERVER 3 / I / | PN(A) VS. PSN(A) ASSUMING Me: 4 0.7 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 I.C PN (A) FIG. 9. fA) VS. PN(A) FOR OBSERVER 3 IN THE SECOND EXPECTED-VALUE EXPERIMENT. 26A) VS N(A SIN

S9-9-6 W3P 8*1-~9-V Z9ZZ 1.0 --------------- 0.9I 0.8 0.5 z 0.7.. OBSERVER 4 3II. y7 PN(A) VS. PSN(A) ASSUMING = 4 0.2 MSN -MN WHERE d'z ~ N 0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 08 0.9 1.0 PN (A) FIG. 10. FP4A) VS. PN(A) FOR OBSERVER 4 IN THE SECOND EXPECTED-VALUE EXPERIMENT. 27

Announced Optimum Observer 1 Observer 2 Observer 3 Observer 4 Day | PN(A) PN(A) PSN(A) PN(A) PSN(A) PN(A) PSN(A) PN(A) PSN(A) 1 1.00.23.32.77.23.84.18.90.18.77 2.50.42.50.88.25.95.12.88.35.83 3.25.58.65.91.41.93.33.85.49.87 4.16.68.78.96.66.95.59.94.74.93 5 2.00.12.05.63.11.80.01.65.17.71 6 4.o0.05.06.79.03.75.02.64.12.64 [r 7 1.50.15.20.85.19.90.15.78.14.76 8 6.00.02.02.82.02.73 1 — -.6.64 9 8.00.00.02.64.00.48.00.45.00.43 10 2.50.09 ----.04.71.08.73.05.56 11.75.28.19.89.18.87.14.84.34.84 12.75.28.29.84.25.89.24.87.34.86 13 8.00.00.03.64.00.59.02.56.04.43 TABLE 4 DATA FROM 4 OBSERVERS IN SECOND EXPECTED-VALUE EXPERIMENT

- ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN n Table 4 labeled "Announced Optimum PN(A)" will be discussed later in this secion of the paper. It may be seen in Figures 7 through 10 that the experimentally determined oints are reasonably well fitted by the type of ROC curve predicted by the theory ubscribed to here. It is equally apparent that points are not well fitted by a traight line intersecting the point PN(A) = PSN(A) = 1. It is important to note, n this connection, that the theoretical ROC curves plotted in Figures 7 through 10.re not exactly the same as those plotted in Figure 1. The exact form of the theoetical ROC curves portrayed in Figure 1, as pointed out in Reference 25, is dependnt upon the assumption that the distributions of N and S + N are Gaussian and of qual variance. It was apparent at the time of writing of Reference 25 that the,ssumption of equal variance was not entirely adequate; the nature of the variance.sswuption, however, was not critical for the purposes of that paper, and the equalariance assumption was accepted to facilitate analysis. The theoretical curves hown here in Figures 7 through 10, which are fitted quite well by the data ploted there, were drawn under the assumption that the ratio of the increment in the:an to the increment in the standard deviation is equal to 4. The implication for nalysis procedures of a dependence between signal strength and variance is disussed in the paper on methods (Ref. 20)..3 The Approach to the Optimum Behavior The fact that the data obtained in the two Expected-Value experiments is itted more adequately by the type of ROC curve consistent with the theory of sigal detectability than by the straight-line ROC curve consistent with the fixed hreshold notion implies that the human observer can vary his criterion. The uestion remains as to how closely he approaches the optimum criterion. --- 29

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN - To establish the applicability of the theory of signal detectability, it is necessary to demonstrate only that the observer's operating level is some monotonic function of P. When sampling error is taken into account, the restriction is more lenient; in this case, it is sufficient to demonstrate a significant correlation between the observer's operating level and P. It is interesting, however, to determine how closely the observer's operating level approaches that specified by P. In the first experiment, the observers were informed of the various values taken on by the elements of the P-equation (see page 6); i.e., the a priori probabilities, the values and costs. They were not made aware of the way in whict these elements determine a value of P, nor did they know that the single number, 3, taken in conjunction with an ROC curve, yields the optimum false-alarm rate. They were informed, after each experimental session, of their false-alarm rate, tl proportion of correct detections of each of the four signal intensities, and the payoff. The correlations obtained in this study between f and false-alarm rate indicate that these observers tended toward the optimum setting of the criterion. For the three observers, the rank-order correlations were.70,.46, and.71; a correlation of.68 is significant at the.01 level of confidence. The first study demonstrated that the human observer quite naturally adjusts his criterion in a way approaching optimum. With this as background, the second experiment was performed with the observers having a fairly complete knowledge of its purposes. The second study attempted to determine how closely the observers could approximate the optimum false-alarm rate, as specified by P, given a knowledge of it. Hence the column in Table 4 labelled "Announced Optimum PN(A)". The experimenter's ability to announce a value of PN(A) approaching the optimum value, before the ROC curves of the observers had been determined directly 30

- ENGINEERING RESEARCH INSTITUTE ~ UNIVERSITY OF MICHIGAN apends upon the fact that, as reported in references 22 and 25, forced-choice and as-no response procedures yield consistent values of d'. The observers in the acond study, like those in the first, had been trained under the forced-choice rocedure, so estimates of d' were available. A comparison of the column in Table 4 labelled "Announced Optimum PN(A)" ad the columns containing the PN(A) obtained from each observer indicate how well he observers reproduced the value announced as optimum. The rank-order correlaions between the announced optimum and obtained values of PN(A) are.94,.97,.86, nd.98. Two of the observers served in twelve sessions; the other two served in hirteen sessions. The rank-order coefficient associated with a probability of 01, given twelve pairs of measures, is.68. In this study, the observers were nformed of their proportion of false alarms after each group of fifty trials. he study of the Neyman-Pearson Observer, reported in the next section of this aper, provides additional evidence for the rather remarkable ability of the obserer to reproduce a given false-alarm rate. 3. THE NEYMAN-PEARSON OBSERVER.1 The Approach to the Announced Optimum Value of PN(A) A different set of four observers served in this experiment. The obserers were informed of the a priori probability of signal occurrence (P(SN) =.72 as held throughout the experiment), but instead of operating in terms of values nd costs, they attempted to satisfy a restriction placed on the proportion of alse alarms. The restriction on the proportion of false alarms took the form of a tipulation for the observers of the acceptable number of "yes" responses to the ourteen "no-signal" presentations in each block of fifty presentations. The 31

ENGINEERING RESEARCH INSTITUTE ~ UNIVERSITY OF MICHIGAN observers were instructed to respond positively to approximately n or n + 1 of th fourteen "no-signal" presentations (n, for the four successive conditions of the experiment, equalled 3, 0, 6, and 9 respectively), so that any proportion of fals alarms, across several blocks of fifty presentations, within a given range of.O0 satisfied the restriction. A given restriction on PN(A) was effective across eighteen blocks of fifty trials. There were then four conditions, each with a different acceptable range for PN(A); thus, the primary data consist of four valu of PN(A) for each observer, with each of these four values based on 252 "no-signa presentations. The acceptable range for PN(A) for the four conditions are shown as column headings in Table 5; the false-alarm rates obtained from the four obser vers appear as cell entries. Note that the largest deviation from the range announced as acceptable is.04. These data, then, also suggest that the observer i able to vary, and quite precisely, the cutoff point on the continuum of observations. It is true that the data presented in the previous paragraph, to the extent that they were reported there, could have been obtained if the threshold concept were valid. If the observer were given immediate knowledge of correctnes of response, any false-alarm rate could be approximated, for example, by saying "yes" until the given proportion of false alarms was achieved and then saying "no on the rest of the presentations. This would entail, however, a severe depressio: of d'. In this study, the observers were given knowledge of correctness of response only after each block of 50 presentations, and the values of d' were not depressed. 3.2 Other Analyses of the Data of this Study In this study, twelve values of signal intensity were used, in addition to the "blank" or "zero-intensity signal". This rather large number of signal 32

TABLE 5 -- NEYMAN - PEARSON OBSERVER Restriction on.21-.29 0.0-.07.43-.50.64-.71 PN(A) Observer 1.28.03.42.74 2.19.02.42.61 3.21.05.39.63 4.22.02.40.60 33

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN values was employed in an attempt to define more adequately the shape of the psychophysical curve. The results of the analysis of curve shape will be discuss in the paper dealing with psychophysical methods (Ref. 20). The nature of the ROC curves resulting from this study cannot be adequately determined since only four points were obtained for each curve; that is, for each value of d' or signal intensity. Neither can the degree of correlation between PN(A) and calculated threshold be estimated since only four pairs of measures were obtained. Thus, the results of this study could not stand alone as evidence for the existence of a variable criterion, or stated more generally, as evidence that the observer's decision function is a monotonic function of likelihood ratio. It is believed, however, that the other four experiments reported herein are sufficient to establish this point. 4. THE A POSTERIORI PROBABILITY OBSERVER The same four observers who served in the second Expected-Value experiment served in this study of the ability of the human observer to report a posteriori probability. In this experiment, P(SN) =.50. The task posed for the observers was to place each observation in one of six categories of a posteriori probability. Here again, the ability to order the observations down into the noise is required. One analysis of the data is reported in Table 6. The categories of a posteriori probability with which the observers worked head the columns. The boundaries of the categories were chosen in conference with the observers; they believed that they would be able to operate reasonably with this particular schem The cell entries show the proportion of the observations placed in a given catego that were, in fact, observations of signal-plus-noise; note that each of the 34

TABLE 6 -- A POSTERIORI PROBABILITY OBSERVER Announced Categories 0.0-.04.05 -.19.20 -.39.40 - 59.60 -.79 80 - 1.00.20.28.45.49.76.97 Observer 1.23.24.42.46.84.95.20.23.28.61.77.95.11.28.47.70.92 1.00 Observer. Ujb 2.14.16.50.86.96 1.00.18.19.38.64.91.96.16.31.76.56.65.94 Observer 3.22.38.34.24.70.92.. 2.21.42.43.36.82.26.71.88 1.00 1.00 1.00 Observer 4.27 ~77.91 1.00 1.00 1.00 ~ 33 ~.53 ~75.78.95.95

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN - observers served in three experimental sessions. Thus, the entry in the upper left-hand corner indicates that 20 percent of the observations placed in the lowest probability category by Observer 1, in the first session, were observations o signal-plus-noise; 28 percent of the observation placed in the next category were observations of signal-plus-noise, and so forth. In nine of twelve cases, i.e., in nine of the twelve rows of Table 6, a rank-order correlation of unity exists between the estimated a posteriori probability and the relative frequency of obse vations arising from signal-plus-noise, indicating an ordering of observations. Each of these correlations has an associated probability of less than.05. All of the three cases showing a correlation of less than unity are attributable to a single observer, Observer 3. 4.1 The Relationship between d(x) and t(x) It is possible, using the data from this experiment, to determine the relation existing between the observer's decision function, d(x) and likelihoodratio, 2(x). For this experiment, with P(SN) =.50, the equation for a posterior probability, given in the introductory section of this paper, reduces to (x) Px(SN) = () +. Since PN(A) and PN(A) can be estimated for the boundaries o the six categories, d' can be determined. A knowledge of PN(A) (relative to each boundary) and d' allows a graphic determination of the value of l(x) correspondin to each boundary since the critical value of l(x) for a given value of PN(A) equals the slope of the ROC curve at the point which corresponds to that value of PN(A). Thus Px(SN) = (x) can be determined for each operating level t(x) + 1 employed by the observer. The values of 2(x) obtained in this way for the l(x) + 1 observer will correspond directly to the probability values marking off the categories - if the observer is operating with a decision function, d(x), that i equal to l(x), and according to the optimum relation between Px(SN) and 2(x). ----------------------—. 36

- ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN Figures 11, 12, 13, and 14 show plots of the probability values corresponding to l(x) )oundary categories versus Px(SN) = (x) determined from the data for the,o2 1(x\ ) + 1-.our observers. Observers 1 and 2 appear to be operating with d(x) similar to!(x) and approximately according to knowledge of the optimum relation between'X(SN) and l(x). This result should probably not be generalized beyond the conLition of P(SN) = P(N) =.50. It should be remembered, however, that the obserrers were exposed to this task for only fourl experimental sessions, with feedback Lfter each session limited to the information reported in Table 6. With respect;o Observers 3 and 4, it is not clear whether they are operating with a d(x) iuite unlike 2(x), or with an imperfect approximation to the optimum relation )etween PX(SN) and d(x), or both. It is possible to determine more exactly the relation between!(x) and L(x) for the four observers. The observer's decision function, d(x), can be set qual to e(x) and the value of w determined. Then the cutoff values of PX(SN) w x ised by the observers are described by (xx) Px(SN) = w)' or Px(SN) = (, and w = 2(x) 1-P x Isa7 2 I(x) + w p w rhere p is set equal to the boundaries of the categories. One interesting question.s whether the weights, w's, are constant for each observer. The values of w lorresponding to each category boundary, for each session and for each observer, Lre given in Table 7... Observers 1, 3, and 4 reported a posteriori probability in a single session preceding th t three sessions reported in Table 6. Since different categories were used, this session was considered as practice, and the data excluded from the analysis. 37

tGS-I~-ZI W3r ILI-~9-V Z9ZZ * DAY I o DAY 2 __ bJA DAY 3 o 0.8 z 0 8Q A 0 z__ 0 0 0. 2 0.4 0.6 0.2!.0 0 0 x 0 o0.2 0.4 0.6 0.8 1.0 Px(SN) = + FIG. II. OBJECTIVE PROBABILITY VS. SUBJECTIVE PROBABILITY FOR OBSERVER I, 38

S-I-ZI W3r ZI-C9-V Z9zz 1.0 * DAY I o DAY 2 A DAY 3 0) 0.8 z > 0.6 ~0.2 /jJ 8 0.2 0.0.8__ U3 i~z3 ao o 0.2 0 0.2 0.4 0.6 0.8 1.0 I (x) PX(SN)= _(x)+! FIG. 12. OBJECTIVE PROBABILITY VS. SUBJECTIVE PROBABILITY FOR OBSERVER 2. 39

s -1 -ZI I 3r 21-LI 9-Vt Z9ZZ 1.0 * DAY I o DAY 2 ___ A DAY 3 0.8 z 0 0 0.6 2 0.4 ----------- z 0 a0.2 z /010/ 000 P (^N) - -^ (X) P (SN)...2(x.+ FIG. 13. OBJECTIVE PROBABILITY VS. SUBJECTIVE PROBABILITY FOR OBSERVER 3. 40

sg-I~-wzi Per Zi-29-v zg9Zz 1.0 * DAY I o DAY 2 A DAY 3 U) w 0.8 LJ / 3 0.6 o n, 0 0.6 0 I — cr' 0 0.2__________ 0 0.2 0.4 0.6 0.8 1.0 /- (x ) Px (SN):(x)+i FIG. 14. OBJECTIVE PROBABILITY VS. SUBJECTIVE PROBABILITY FOR OBSERVER 4. 41

TABLE 7 -- THE REIATIONSHIP BETWEEN d(x) and i(x)+ Category Boundaries Observer and Day ~.05.20.40.60.80 Day 1 2.9 1.7 1.3 2.2 --- Obs. 1 Day 2 3.2 2.0 1.4 2.2 - Day 3 2.5 1.6 1.3 1.4 - Day 1 3.2 2.2 2.3 4.0 - Obs. 2 Day 2. 95 2.0 3.5. Day 3 1.9 1.7 1.9 2.7 - Day 1 5.3 3.8 1.7 1.2 1.4 Obs. 3 Day 2 6.4 2.6 1.4 1.2 1.0 Day 3 4.0 1.6..4.3 Day 1 28.5 20.0 --- Obs. 4 Day 2 34.2 36.0 -- -- Day 3 30.4 11.6 6.9 4.7 --- The weights for some of the higher categories are indeterminate since the critical value of l(x) approaches infinity as PN(A) approaches 0.0. 42

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN To determine the degree to which w approximates a constant for each ession and each observer, a given fixed value of w can be chosen for each obserer, and used to determine Px(SN) = (x). If the "true" values of w are l(x) + w pproximately constant for a given observer, then a plot of the points, representng category boundaries versus P(SN) (with w fixed), should be adeX I. X) + wV( uately fitted by a straight line intersecting the points (0,0) and (1.00, 1.00). hat is, each of the lines in Figures 11 through 14 should correct to the diagonal. his, however, is not the result. Using fixed values of w resulted in a plot of oints for Observers 1 and 2 that are only slightly better fitted by the diagonal ine than are the points shown in Figures 11 and 12. A fixed value of w greatly rnproves the correspondence of the diagonal line and the plotted points for bserver 4 - for Days 1 and 2. The correspondence for Observer 3 is not improved oticeably by the transformation. It may be noted, however, that there is the suggestion (in Table 7) of consistent pattern between "subjective probability" and "objective probability" hat is, of a pattern in the relation between w and the category boundaries. The's for the last two sessions for Observer 1 show the same rank-order. For Obserer 3, the values of w rank from left to right for each of the three sessions with single exception. 5. THE SECOND-CHOICE EXPERIMENT In this experiment, a variation of the forced-choice method of response as employed to demonstrate the ability of the observer to order observations.l This experiment was suggested by R. Z. Norman, formerly a member of the Electronic Defense Group, now at Princeton University. 43

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN In the version of the forced-choice method most commonly used, the observer knows that on each trial the signal will occur in one of four short, successive time intervals, and he is forced to choose in which of these intervals he believes the signal occurred. If the observer can order observations, then to behave optimal] he must select the interval with the greatest associated observation. If the observer behaves optimally, then the probability that a correct answer will result for a given value of d', for the four-choice or four-interval situation, is given by the equation: P(c) = [F(x)] g(x)dx (1) -cOO where F(x) is the area of the noise distribution and g(x) is the ordinate of thN signal-plus-noise distribution. The value of P(c) corresponding to each value of d' is shown by the middle curve of Fig. 15. This curve is plotted under the assumption that the distributions of N and S + N are Gaussian and of equal variance 5.1 The Rationale for the Second-Choice Experiment Consider the situation where the observer is required to indicate a second choice as well as a first choice. What is the probability of a correct second choice on those trials on which the first choice is incorrect? The conventional notion of a threshold -. as pointed out above and, in more detail, in Reference 25 - is essentially that the mechanism of detection is one that triggers when the observation exceeds a critical amount, and loses all discrimination among observations falling short of this amount. For the (four-choice) forced-choice situation where the observer is required to indicate a second choice as well as a first choice, this conception of the mechanism leads to the prediction that, when the first choice is incorrect, the probability that the second choice will be correct is.33. According to this 4

- ENGINEERING RESEARCH INSTITUTE ~ UNIVERSITY OF MICHIGAN iew, in other words, the second choice is made from among three intervals on a hance basis. On the other hand, according to the conception of the human observer as perating in terms of likelihood ratio, the observer is capable of ordering the our observations associated with the four intervals. If this is the case, the Proportion of correct second choices, on those trials in which an incorrect first;hoice is made, should be greater than.33. The relationship between this pre-.icted probability and d' is given by the expression 3 F(x)] [1 - F(x)] g(x) dx (2) i-,, (2) 1 - [F(x)] g(x) dx -00 there the symbols have the same meaning as in Equation 1 above. This relationship.s plotted in Figure 15 under the assumption that the distribution of N and S + N Lre Gaussian and of equal variance.;.2 Results Data were collected from four observers; two of them had served preiously in the second Expected-Value experiment whereas the other two had only'eceived routine forced-choice training. Each of the observers served in three!xperimental sessions. Each session included 150 trials in which both a first Lnd second choice were required. The resulting twelve proportions of correct;econd choices are plotted against d' in Figure 15. The values of d' were deterlined by using the proportions of correct first choices as estimates of P(c) and ~eading the corresponding values of d' from the middle curve of Fig. 15. Although single value of signal intensity was used, the values of d' differed sufficiently rom one observer to another to provide an indication of the congruence of the I-5

g -ez-f, W 3r 021-~9-V 0O61 1.0 0.9 - CHOICE 3- CHOIE d CHOICE 0,7 0.6 O0.5 - 0.54 J- -------------------------— 46 \y~~~~~~~~/ ~FIXED CRITERION:O.33 o.3 0 __~-...... — --.-... — -. 0.2 - --- 0.1 i i 0 I 2 3 4 5 6 d FIG. 15. THE SECOND-CHOICE DATA AND PREDICTIONS FOR THE SECOND-CHOICE SITUATION UNDER THE ASSUMPTION THAT THE N AND S+N DISTRIBUTIONS ARE GAUSSIAN AND OF EQUAL VARIANCE. 46

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN Lata with the predicted functions. Additional variance in the estimates of d''esulted from the fact that, for two observers, a constant distance from the sigial was not maintained. (The function predicted by the theory of signal detec-;ability for the proportion of correct first choices in a three-choice, or three.nterval, situation is included in Fig. 15 to indicate that this function is not;he same as the predicted function of the probability of a correct second choice,;iven an incorrect first choice, for the four-choice situation). A systematic deviation of the data from a proportion of.33 clearly!xists. Considering the data of the four observers combined, the proportion of:orrect second choices is.46. The deviation of this obtained proportion from 33 is highly significant, the X' obtained (43.66) is more than twice the (2(19.0) associated with a probability of.00001. Two control conditions aid in interpreting these data. The first of nese allowed for the possibility that requiring the observer to make a second hoice might depress his first-choice performance. During the experiment, blocks if 50 trials in which only a first choice was required were alternated with blocks f 50 trials in which both a first and second choice were required. Pooling the ata from the four observers, the proportions of correct first choices for the wo conditions are.650 and.651, a difference that is obviously not significant. A preliminary experiment in which data were obtained from a single oberver for five values of signal intensity also serves as a control. This experient substantiates the predicted correlation between the probability of a correct econd choice and signal intensity that derives from the theory of signal detecability. 150 observations were made at each value of signal intensity. The elative frequencies of correct second choices for the lowest four values of sigal intensity were, in increasing order of signal intensity, 26/117 (.22), 47

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN 33/95 (.35), 30/75 (.40), and 20/30 (.67). For the highest value of signal inte sity, none of five second choices were correct. Thus, the proportion of correct second choices is seen to be correlated with a physical measure of signal intensity as well as with the theoretical measure (d') - this eliminates the possibility that the correlation found with a constant value of signal intensity, involving d' as one of the variables (Figure 15), is an artifact of theoretical manipulation. The second-choice data, then, demonstrate clearly the untenabilit of the assumption of a fixed criterion or threshold. It may be seen from Fig. 15 that second-choice data also deviate systematically from the predicted function derived from the theory of signal detectability. This discrepancy results from the inadequacy of the assumption - of equal variance of the noise and signal-plus-noise distributions - upon which the predicted functions in Fig. 15 are based. It was pointed out above and in Refer. ence 25 that the equal-variance assumption was accepted in the early stages of data collection in order to facilitate analysis, in spite of existing indication, of its inadequacy. It was also pointed out above that more recent data on the Expected-Value Observer (see Figures 7-10 and accompanying text) indicate that a valid assumption would be the assumption that the ratio of the increment in the mean to the increment in the standard deviation, M, is equal to 4. Figure 16 AC shows the second-choice data and the predicted four-choice and second-choice cur derived from the theory of signal detectability under the assumption that AM = 4. In view of the variance associated with each of the points (each firstchoice d' was estimated on the basis of 300 observations and each second-choice proportion on less than 100 observations) the congruence of the data and the predicted function shown in Fig. 16 is quite remarkable. 48

gS-8-~I W3P ~81-C"9-v 9ZZ 1.0 r, 4-CHOICE 0.9.... —-_ —---- 0.8 / ~ 0.6 a L 0.6 2 na CHOICE 0.5 0.4. —-- FIXED CRITERION: 0.33 0.3 /* 0.2 0.1 0 0 1 2 3 4 5 6 d' FIG. 16. THE SECOND-CHOICE DATA AND PREDICTIONS FOR THE SECOND-CHOICE SITUATION UNDER THE ASSUMPTION THAT 4. Ao49

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN 5.3 A Note on the Variance Assumption The particular assumption made in this paper about the variance of the noj and signal-plus-noise distributions, namely that M = 4, needs qualification in t respects. First, it is very likely specific to the experimental conditions employed. Second, it is to be regarded as only provisionally applicable to the present data. If the variance of these sampling distributions is a function of sample size, then it may be presumed that their variances are different for different signal durations, or observation times, and for signals of different sizes. If the variance of the noise and signal-plus-noise distributions decreases with increases in signal size and duration, then an assumption concerning the increase i variance with increasing signal intensity is applicable only to data collected using a single duration and size of signal. Since the various experiments reported here employed identical signal conditions, it was possible to assess the adequacy of a single variance assumption for different forms of data, for "yes-nc data in Section 2 and for forced-choice data in Section 5. Although positive results were obtained from this check of internal consistency, it should not be in ferred that the particular assumption will describe the results of experiments in volving different physical parameters. AM Also, as indicated, that the assumption that = 4 derives from the data reported here is advanced with certain provisions. Other assumptions have not be thoroughly explored. It may be that ratios equal to certain other constants will fit the reported data even better, and that several other constant ratios will fi the data as well. The sensitivity of the assumed ratio to the existing data has not as yet been determined. It is likely that more precise data is required for the purpose of determining the relative adequacy of different variance assumption These problems are presently being explored; the results will be reported in the paper concerned with methods of data analysis (Ref.20), 50

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN 6. DISCUSSION The general conclusion drawn from the experimental results reported )ove is that the model provided by the theory of signal detectability (Refs. i, 16) or more generally, by the theory of statistical decision (Refs. 13, 14, 27, 3) is applicable to the detection behavior of the human observer. This model will:oduce data like those observed. This type of model has come to be called a "computer model". The term computer", in this connection, is meant very generally; it includes anything that,ocesses information in a precisely defined way. Quastler's discussion of the Lture of the computer model is pertinent here. "The computer model may be a system of equations. It may be a black-box diagram with boxes labelled'receiver','memory','transducer','decision', etc. It may be a piece of hardware. It may have many or few components; it may be determinate or stochastic. Neither the size nor the type nor the physical nature of the model matter. All that does matter is that it should serve as a framework to organize past and future experience" (Ref. 17). became clear to the authors, as the series of experiments reported above was,ing carried out, that the model described served admirably the function of'ganizing past and future experience. The block-diagram representation of the xdel of the Expected-Value Observer, after Quastler (Ref. 17), is shown in.gure 17. P(SN), V'S a K'S CRITERION COMPUTER A(P) SIGNAL RECEIVER OBSERVATION (X).(X). (X) DECISION A OR CA COMPUTER COMPUTER FIG. 17. BLOCK DIAGRAM OF THE EXPECTED-VALUE OBSERVER 51

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN The same diagram applies to the Neyman-Pearson Observer when the input to the criterion computer (P(SN), V's, and K's) is replaced by PN(A(p)) = k. In the case of the A posteriori Observer, there is, of course, no criterion com puter, and the decision computer is replaced by a computer which makes the trans formation from l(x) to Px(SN), having l(x) and P(SN) as input and PX(SN) as output. To represent the forced-choice situation, the criterion computer is eliminated, and the decision computer merely selects the greatest of the input likeli hood ratios as its output. Each of the five experiments reported above demonstrates that the huma observer operates with information in the form of likelihood ratio, and tends toward optimum behavior. These experiments provide convincing evidence of the applicability of the proposed model to the problems of visual detection. The re vance of the model is also supported by its congruence with auditory data (Ref. 26). There is, however, still another imposing reason for treating sensory prob lems in terms of the model, namely, that the model provides a unification of the data obtained with forced-choice and yes-no methods of response. If psychophysil data is collected and analyzed in accordance with this model, performance in the forced-choice situation can be predicted from yes-no data, and vice versa. Evidence presented elsewhere (Refs. 22, 25) shows that the estimates of d' from the two response procedures are highly consistent. It is interesting to note, parenthetically, that the present account i; not among the first to model psychophysical theory after developments in the the( of statistical decision. Fechner, the founder of psychophysics, was appreciably influenced by Benoulli who first suggested computing expectations in terms of satisfaction units. As Boring (Ref. 5) relates the story, Bernoulli's interest in games of chance led him to formulate the concept of "mental fortune"; a change 52

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN a mental fortune he believed to vary with the ratio that the change in physical ortune has to the total fortune. This mathematical relation between mental and hysical terms was the relation that Fechner sought to establish' ith his pyc:hohysics..1 The Possibility of a Threshold Theory It will be well to consider again, after the presentation of the data, he possible validity of a theory incorporating the threshold concept. It is lear that the assumption of a threshold as previously conceived, one that is very arely exceeded by noise alone, is not reasonable. At this time, however, the xistence of a threshold well down into the noise distribution cannot be disredited. To take a single example, the data, representing values of PN(A) above point between.10 and.30, that are fitted to the R.O.C. curves in Figures 7 trough 10 may be adequately fitted by a straight line through PSN(A) = PN(A) =.0; these data, then, do not preclude postulation of a threshold in the neighborood of the mean of the noise distribution. Further analyses of the second-choice ata, in relation to thresholds at various levels in the noise distribution, are eing undertaken and will be reported elsewhere (Ref. 20). It should be noted, however, that determination of the level of the oise distribution at which a threshold may possibly exist is neither critical or useful. A threshold at such a level is not a readily workable concept. The rimary virtue of a threshold that is rarely exceeded by noise alone is that it acilitates mathematical treatment of the data, chiefly by being consistent with he usual correction for chance. It has been demonstrated above, however, that athematical manipulations must not involve assumptions incompatible with a noise istribution much of which exceeds a threshold, if a threshold is to be postuated; at this point then, adhering to a threshold concept complicates the ---------. ~~~~~53

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN mathematics. As a matter of fact, a threshold at a level well within the range of the noise distribution, is, for all practical purposes, not measurable. The forced-choice methodology is a case in point; the observer conveys less information than he is capable of conveying if only a first choice is required. That t second choice contains a significant amount of information has been demonstrated it is not unlikely that the third choice will convey information. Thus it is very difficult for an experimenter to determine when enough information has beer extracted from forced choices to yield a sufficiently low estimate of the thresl old. In addition, the existence of such a threshold is of no consequence to the application of the theory proposed here; to take an example, "yes-no" data resulting from a suprathreshold operating level depends on the operating level but is completely independent of the threshold value. 6.2 Some Implications of the Proposed Model for Psychological Theory The applicability of the model provided by the theory of signal detect bility stands in opposition to the view that the so-called sensory phenomena are independent of control by general psychological variables, a view that is consis tent with the concept of the threshold. The theory built upon this model takes into account the influence on detection behavior of "non-sensory" central determinants. In conventional theory, the decision concerning the existence of a sig nal is assumed to depend entirely upon the threshold being exceeded, and the threshold level is assumed to be independent of control by other variables that might influence the attitude or set of the observer. A successful application o the theory of signal detectability to psychophysical data leads to the replacement of the threshold concept by a concept of criterion range of acceptance; the way in which the control of this range is conceptualized acknowledges the relevance of variables that influence set. I —- ~ 54

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN In the framework of a statistical theory, it is necessary to make an asmption which permits the definition of behavior, or the prescription of predicd behavior. The concept of the threshold has served this purpose. The assumpon that the threshold is relatively invariant permits making predictions which n be tested experimentally. In the theory presented here, where the position of e cutoff between acceptance and rejection of the existence of a signal (the oper. ing level) is assumed to be under the control of the observer, it is necessary define the method of control exerted by the observer on the operating level. e assumption that the observer tends toward the optimum behavior provides the cessary definition. Thus, in the typical yes-no experiment, the observer may be nceived of as regulating the operating level in terms of a priori probabilities d the values and costs associated with the various types of correct and incorrect swers, in such a way as to maximize the total expected gain. (It should, perps, be pointed out that these values and costs do exist in the typical yes-no periment, whether or not they are explicitly translated into numerical values). the forced-choice experiment, optimum behavior requires that no operating level assumed, and that the interval with the greatest associated observation be sected. Although there has been a general discontent among psychologists with the ncept of a fixed operating level or threshold, there has not been advanced preously a way of defining the mode of control exerted by the individual over a riable operating level. It is the chief virtue of the model provided by the eory of signal detectability that it specifies operating level variability. 3 Some Implications for Practice The results of these experiments, of the Expected-Value experiments in rticular, give an account for Blackwell's (Ref. 3) finding that forced-choice ta are more reliable than yes-no data. In the yes-no experimental setting, when e usual caution against making false-alarm responses is included, the operating 55 -

ENGINEERING RESEARCH INSTITUTE * UNIVERSITY OF MICHIGAN level may vary over a wide range, with the variation having no direct reflection in the data. False-alarm rates of.01,.001, and.0001, for example, are not d: criminable in an experimentally feasible number of observations. This fact may also account for the failure to detect previously the operation of a mechanism with a variable operating level. The results presented above account also for the often-reportedl find: that the forced-choice procedure yields lower calculated thresholds than does t] yes-no procedure. Conventionally, observers are cautioned against making false. alarm responses in yes-no experiments. It is very likely that the stigma attacl to "hallucinating" serves to depress PN(A) in those test settings not including explicit warning to avoid false alarms. The inverse relationship between PN(A) and calculated threshold predicted by the theory of signal detectability is dis. cussed in Reference 25. Data substantiating this predicted relationship include the correlations reported above and in Reference 25 between PN(A) and calculatec threshold, and that shown in Figure 5 above. Plots of PSN(A) vs. PN(A) which dc not fit a straight line through PSN(A) = PN(A) = 1.00(such as in Figures 7-10) provide another way of saying the same thing. The relationship between d' and PN(A) at "threshold" is described in more detail in the forthcoming paper on methods (Ref. 20). In spite of the view that "sensory" phenomena are peripherally determined, the necessity of assuming a constancy of "set" across people and over tin in psychophysical experiments has been generally accepted. Effecting this constancy is a primary function, at least in the yes-no experiments, of the verbal instructions. The theory of signal detectability and the experiments reported have advanced a check on the assumption of constancy of set with respect to the most important aspect of set in yes-no experiments, namely, the location of the 1. See, for example, Blackwell (Ref. 3) and Goldiamond (Ref. 9). Miller (Ref. reviews earlier literature on "subliminal perception" that is relevant to t1 point. 56

ENGINEERING RESEARCH INSTITUTE ~ UNIVERSITY OF MICHIGAN toff between acceptance and rejection of the existence of a signal. This rearch has demonstrated that a measurable estimate of PN(A) can be, and should be, oduced in yes-no experiments. By the same token, theoretical support has been ovided for the advisability of using the forced-choice procedure whenever posble. With this technique, the observer is not faced with the problem of locatg and maintaining the stability of an operating level, and thus a source of riance in the data is removed. Since a method for unifying forced-choice and s-no data has been provided (Refs. 22, 25), forced-choice techniques can be used en when an estimate of performance in the yes-no situation is desired. The praccal import of this is that data pertinent to yes-no situations, that is more reable, can be obtained with greater economy. This topic is treated in more deil in the paper on methods (Ref. 20). That the condition of the organism affects perception has been demonrated previous to the studies reported here. The theory of Ames and his corkers (Ref. 1), the "new look" theory of Bruner and Postman (Ref. 6), and the.eories of Hebb (Ref. 11), Brunswik (Ref. 7), and Woodworth (Ref. 29) have taken to account the effect of "non-sensory" central determinants. The present acunt, however, goes beyond the stage of demonstration; it provides operational d theoretical specifications of the "conditions". The present account, then, tisfies the desiderata discussed by Graham (Ref. 10): the conditions of the ganism are specified at other than the conversational level; the conditions are fined in the theory and anchored to operations at both ends. Since the variables ich determine these conditions are expressed quantitatively, the quantification an important group of instruction stimuli has been achieved. Said another way, is research proceeds a step in the direction of specifying conditions of the ganism, due to instructions, as parameters of observable stimulus-response lations. 57

REFERENCES 1. Ames, A., Jr., "Visual Perception and the Rotating Trapezoidal Window," Psycl Monogr., 65, No. 7, 1951. 2. Birdsall, T. G., "An Application of Game Theory to Signal Detectability," Tec Rpt. No. 20, Electronic Defense Group, University of Michigan, Ann Arbor, Mi( gan, 1953. 3. Blackwell, H. R., Psychophysical Thresholds: Experimental Studies of Methodc Measurement. Ann Arbor, University of Michigan Press, 1953 (Eng. Res. Inst. Bulletin No. 36). 4. Blackwell, H. R., Pritchard, B. S., and Ohmart, T. G., "Automatic Apparatus i Stimulus Presentation and Recording in Visual Threshold Experiments," J. Opt. Amer., 1954, 44, 322-326. 5. Boring, E. G., A History of Experimental Psychology. New York: Appleton-Cel Crafts, 1950 (2nd edition). 6. Bruner, J. A., in Blake, R. R. and Ramsey, G. V. (Eds.) Perception: An Apprc to Personality. New York: Ronald Press, 1951. 7. Brunswik, E., Systematic and Representative Design of Psychological Experimer Berkeley: University of California Press, 1949. 8. Fox, W. C., "Signal Detectability: A Unified Description of Statistical Metl Employing Fixed and Sequential Observation Processes," Tech. Rpt. No. 19, Elc tronic Defense Group, University of Michigan, Ann Arbor, 1953. 9. Goldiamond, I., "The Relation of Subliminal Perception to Forced Choice and Psychophysical Judgements, Simultaneously Obtained," Amer. Psychol., 9, No. E 1954, (abstract). 10. Graham, C. H. "Visual Perception," in S. S. Stevens (ed.) Handbook of Experin Psychology. New York: Wiley, 1951. 11. Hebb, D. O., The Organization of Behavior. New York:.iley, 1949. 12. Miller, J. G., Unconsciousness. New York: Wiley, 1942. 13. Neyman, J., "Basic Ideas and Some Recent Results of the Theory of Testing StE tistical Hypotheses," J. Roy. Stat. Soc., CV, Part IV, 1942. 14. Neyman, J., and Pearson, E. S., "On the Problem of the Most Efficient Tests c Statistical Hypotheses," Phil. Trans. Roy. Soc., 231, Series A, 1933. 15. Peterson, W. W., and Birdsall, T. G., "The Theory of Signal Detectability," ] Rpt. No. 13, Electronic Defense Group University of Michigan, Ann Arbor, 195] 58

Peterson, W. W., Birdsall, T. G. and Fox, W. C., "The Theory of Signal Detectability," in the Transactions of the 1954 Symposium on Information Theory held at Massachusetts Institute of Technology, Cambridge, Massachusetts, September 15-17, 1954. Published by the Institute of Radio Engineers, 1 E. 79th St., New York 21, N. Y. Quastler, H., Notes on the Concluding Discussion of the "Symposium on Methodology in the Estimation of Information Flow," held at the University of Illinois, Urbana, July 5-9, 1954 (mimeo). Shannon, C. E., "The Mathematical Theory of Communication," Bell System Technical Journal, 1948. Swets, J. A., "An Experimental Comparison of Two Theories of Visual Detection," unpublished doctoral dissertation, University of Michigan, 1954. Tanner, W. P., Jr., Birdsall, T. G., and Swets, J. A., "Psychophysical Methods," Tech. Rpt. No. 42, Electronic Defense Group, University of Michigan, Ann Arbor, 1955 (in preparation). Tanner, W. P., Jr., and Norman, R. Z., "The Human Use of Information. II. Signal Detection for the Case of an Unknown Signal Parameter," in the Transactions of the 1954 Symposium on Information Theory held at Massachusetts Institute of Technology, Cambridge, Massachusetts, September 15-17, 1954. Published by the Institute of Radio Engineers, 1 E. 79th St., New York 21, N. Y. Tanner, W. P., Jr., and Swets, J. A., "A New Theory of Visual Detection," Tech. Rpt. No. 18, Electronic Defense Group, University of Michigan, Ann Arbor, 1953. Tanner, W. P., Jr., and Swets, J. A., "A Psychophysical Application of the Theory of Signal Detectability," Electronic Defense Group, Report 1970-5-S reprinted from minutes of Armed Forces NRC Vision Committee Meetings held at Fort Knox, Kentucky, November, 1953. Tanner, W. P., Jr., and Swets, J. A., "The Human Use of Information, I. Signal Detection for the Case of the Signal-Known Exactly," in the Transactions of the 1954 Symposium on Information Theory, held at Massachusetts Institute of Technology, Cambridge, Massachusetts, September, 15-17, 1954. Published by the Institute of Radio Engineers, 1 E. 79th St., New York 21, N. Y. Tanner, W. P., Jr., and Swets, J. A., "A Decision-Making Theory of Visual Detection," Psychological Review, Vol. 61, No. 6, 1954. Tanner, W. P., Jr., Swets, J. A., and Green, D. M., "The General Properties of the Hearing Mechanism," Tech. Rpt. No. 30, Electronic Defense Group, University of Michigan, Ann Arbor, 1955, (in preparation). Wald, A., "A Contribution to the Theory of Statistical Estimation," Amer. Math. Stat., 10, 1939. Wald, A., Statistical Decision Functions. New York: Wiley, 1950. Woodworth, R. S. Psychology. New York: Holt, 1945 (4th Edition). 59

DISTRIBUTION LIST 1 copy Director, Electronic- Research Laboratory Stanford University Stanford, California Attn: Dean Fred Terman 1 copy Commanding General Army Electronic Proving Ground Fort Huachuca, Arizona Attn: Director, Electronic Warfare Department 1 copy Chief, Engineering and Technical Division Office of the Chief Signal Officer Department of the Army Washington 25, D. C. Attn: SIGJM 1 copy Chief, Plans and Operations Division Office of the Chief Signal Officer Washington'25, D. C. Attn: SIGOP-5 1 copy Countermeasures Laboratory Gilfillan Brothers, Inc. 1815 Venice Blvd. Los Angeles 6, California 1 copy Commanding Officer White Sands Signal Corps Agency White Sands Proving Ground Las Cruces, New Mexico Attn: SIGJWS-CM 1 copy Commanding Officer Signal Corps Electronics Research Unit 9560th TSU Mountain View, California 75 copies Transportation Officer, SCEL Evans Signal Laboratory Building No. 42, Belmar, New Jersey FOR - SCEL Accountable Officer Inspect at Destination File No. 22824-PH-54-91(1701) 60

1 copy H. W. Welch, Jr. Engineering Research Institute University of Michigan Ann Arbor, Michigan 1 copy Document Room Willow Run Research Center University of Michigan Willow Run, Michigan 11 copies Electronic Defense Group Project File University of Michigan Ann Arbor, Michigan 1 copy Engineering Research Institute Project File University of Michigan Ann Arbor, Michigan 61