2144-152-T Report of Project MICHIGAN Evaluating Surveillance Systems W. V. Caldwell, C. H. Coombs, and R. M. Thrall June, 1957 The University of Michigan Engineering Research Institute Willow Run Laboratories Willow Run Airport Ypsilanti, Michigan

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE Contract No. DA-36-039 SC-52654 1ii

2144-152 -T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE ABSTRACT The first phases of a long range project for developing models for evaluating intelligence systems in terms of measurable outputs can now be reported. In making such evaluations, the number of correct identifications, number of false alarms, number of displacements, and number of complete misses must be measured, in each of several categories (such as infantry, artillery, armor, supply lines), each of a number of sectors (front line, rear, far rear) and in each of several situations (such as attack or defense, good or poor mobility). The major results to date are (1) the development of experimental techniques, (2) the formulation of an analytical model, (3) the development of analysis techniques for evaluation of parameters in the model in terms of empiricalresults, (4) the preliminary evaluation of parameters (rank order correlations of 0. 8 and higher have been established between some of the analytic predictions and the empirical results), and (5) formulation of several concrete unsolved problems out of the over-all problem. Plans for continued study are discussed in Sections 8. 2 and 8. 3. iii

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE TABLE OF CONTENTS Section Title Page Abstract List of Figures vii List of Tables ix Preface xi 1 General Considerations 1 1.1 Introduction 1 1. 2 Scope of the Model 1 2 The Model 2 2. 1 Sectors and Categories 2 2. 2 The Basic Equation 3 2. 3 Valuation Indices 4 3 Experimental and Measurement Problems 5 3..1 Experimental Design 5 4 The Pretest 6 4.1 Nature of Pretest 6 4. 2 Test Materials 6 4.3 Subjects 6 4. 4 Instructions and Procedure 8 4.5 Results 8 5 Fort Benning Study 10 5. 1 Nature of Fort Benning Study 10 5. 2 Test Materials 10 5.3 Subjects 10 5, 4 Instructions and Procedure 10 5. 5 Agreement between Judges 11 6 Fort Sill Study 12 6. 1 Reason for Fort Sill Study 12 6.2 Subjects 12 6. 3 Agreement Among Judges 12 6.4 Analysis of Judgment Criteria in Artillery 12 Situations 6.5 Relative Weights Implicitly Assigned by 14 Judges 7 Development of the Model 15 7. 1 General Considerations 15 7. 2 Transformation of Empirical Values to 16 Values of d V

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE TABLE OF CONTENTS CONTINUED 8 Concluding Remarks 18 8.1 Summary of Results 18 8. 2 Some Unsolved Problems 18 8. 3 Future Developments 19 Appendix A Methods of Determining Coefficients for Basic 21 Equation Appendix B Definition of r, W, and Pav 23 Appendix C Composition of the Stimuli 25 Appendix D Rankings of Situations by Individual Officers 27 Appendix E Briefing Procedure 31 Appendix F Debriefing Procedure 33 Distribution List 35 vi

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE LIST OF FIGURES Number Title Page 4. 2-1 Actual and Estimated Situations 7 6. 1-1 Plot of Average Rank of Infantry Situations from Ft. Benning with those from Ft. Sill 14 6. 1-2 Plot of Average Rank of Artillery Situations from Ft. Benning with those from Ft. Sill 15 6. 1-3 Plot of Average Rank of Infantry-Artillery Situations from Ft. Benning with those from Ft. Sill 15 7. 1-1 Illustration of the Variables 18 7. 2-1 Comparison of Observed Values with Predicted Values (Infantry situations only) 19 vii

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE LIST OF TABLES Number Title Page 4-1 Infantry Display Only 8 4-2 Artillery Displays Only 9 4-3 Infantry-Artillery Combined 9 4-4 Community of Agreement in Ranking 10 5-1 Agreement of Each Judge's Rank Order with the Average Rank Order of the Remaining Judges 11 5-2 Community of Agreement in Ranking 11 6-1 Agreement of Each Judge's Rank Order with the Average Rank Order of the Remaining Judges 12 6-2 Community of Agreement in Ranking 13 6-3 Ranking of Selected Artillery Situations by Subgroups 13 6-4 Average Experience (Mo. ) 13 6-5 Correlations of Infantry-Only with Infantry-Artillery Combined 13 7-1 Three Possible Criteria for Measuring the Variables 16 7-2 Measures Obtained from One Situation Applying Three 16 Different Criteria 7-3 The Coefficients and Fit of Equation 26 for Three Values of a 17 7-4 The Coefficients and Fit of Equation 27 for Four Values of a 18 ix

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE PREFACE Project MICHIGAN is a Joint Service supported project at the Engineering Research Institute, The University of Michigan. Its general mission is the conduct of research and development of systems and components for combat surveillance. It operates under the cognizance of the Chief Signal Officer, Department of the Army. The aims of Project MICHIGAN are: to supplement the functions of the Technical Services in the research and development of equipment for surveillance, target detection, target location, and data transmission; to make maximum use of the techniques and equipment developed by the Technical Services and to emphasize their ultimate use in the combat surveillance system; and to engage in such research and development as may be found necessary to fill gaps in the existing programs leading to combat surveillance. xi

2144-152- T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE GENERAL CONSIDERATIONS 1.1 INTRODUCTION One of the most difficult aspects of the problem of evaluating a surveillance system is the measureIn both war and peace military leaders are ment of the accuracy and the importance of the inforced to make decisions regardless of whether they formation it provides. consider themselves fully prepared to do so. Although intuition has often been the sole basis for A battlefield-surveillance system includes many choices, much current research is devoted to the components which are sources of information about study of decision processes. Some of this research the enemy, such as aerial photographs, prisoner of concerns statistical and game theoretical techniques war interviews, heat detectors, patrols, and radar. for arriving at decisions under conditions of uncer- The output of each component takes the form of intainty, and other research concentrates ontechniques formation (correct or incorrect) on such items as for reducing the amount of uncertainty. The work the identification of enemy units, their location, and here reported falls primarily into this second cate- their strength. The output of one component results gory. in intelligence which will not in general be identical with that resulting from another Component. For This study is the beginning of a long range at- this and other reasons there arises the difficult probtack on the problem of building a model for evaluat- lem of evaluating a component or even an entire sysing an intelligence system in terms of its total out- tem. puts as presented to the decision making officers (at all levels). The parameters in the model serve There are many criteria which should enter inas measures of the relative importance of various to the comparative evaluation of such systems. types of information and so will aid in preliminary These may be classified under the headings of perevaluation of proposed equipment for gathering in- formance, military feasibility, technical feasibility, formation. It seems clear from what has been done and logistics. Many considerations under these already that a suitable model can be built, although classifications, such as speed, allweather capability, its final form can be developed only after further mobility, and manpower requirements, are objecstudy. As regards theory, analysis has not as yet tively observable and not very difficult to measure. gone far beyond dealing with single components. Much more difficult is the determination of how to Results are encouraging and justify continued re- combine judgments based on these many considersearch. ations to yield an over-all evaluation of a component or a system. The' basic idea of the model is to achieve a good linear approximation for a nonlinear situation by splitting a large problem into many small pieces. The score given to an intelligence picture will be a The model, which is formally presented below, linear combination of the scores assigned to each of was designed to be used in solving problems of the its small pieces. The score assigned to a small following type: given an enemy situation and an espiece must be computable in terms of the basic vari- timate of it, how may one determine (measure) the ables: hits, false alarms, displacements, and mis- merit of the estimate. An estimate may (1) correctses. The model does not require that the score as- ly identify and locate certain enemy units, (2) report signed to each small piece be linear in these basic some nonexistent units, (3) report certain actual variables, but it does seem desirable to keep this units but give an incorrect location for them, and scoring as nearly linear as possible. (4) fail altogether to detect certain units. These are called, respectively, hits, false alarms, displaceA central feature of the model is the use of judg- ments, and misses and each is a basic variable in ments of experienced officers for determining the the model. They must be combined to obtain a parameters in the scoring equations. One of the measure of accuracy that enters into the final "value" main contributions of the work thus far is the devel- of an intelligence estirhate. Each is monotonically opment of procedures for obtaining the necessary related to the value, but its relative contribution is judgments. not known (a priori) For instance, increasing the 1

2144-152- T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE sensitivity of a detector might not only increase hits compared with an active light artillery concentration and decrease misses (which would contribute to value) in the same area, but the reverse is true fifty miles but could also increase false alarms (which dimin- to the rear. ishes value). The value of the detector and its optimum sensitivity, then depends in part on the relative In summary, then, the major aspects of a surimportance of hits, false alarms, displacements, veillance system which will be considered here as and misses. contributing to the value of its intelligence display will be: In addition to constructing parameters which (1) the relative weights of hits, false alarms, measure the relative importance of hits, false displacements, and misses within a given alarms, displacements, and misses for a given cat-. category of information and within a given egory of information such as infantry units or artillery units, there is the problem of measuring the relative importance of information in the several (2) the relative importance of the several catecategories. Such measures are needed to judge be- gories of information within a given sector; tween two systems, one of which is, for example, and superior in detecting personnel and the other of which is superior in detecting motor vehicles. (3) the effect of the nature of the sector upon (1) and (2). Finally, there is another group of factors which It is the purpose of our model to provide analytmust be taken into account in determining the value ic expressions for these relative values. The basic of an intelligence display. This group has to do with parameters of the model are derived from subjecthe character of the terrain and the depth of the sec- tive judgment made by military officers with contors to which the display refers. It is reasonable to siderable combat and staff experience. Consequentexpect that the relative weights of hits, false alarms, ly, a statistical procedure has been set up for condisplacements, and misses within a category of in- verting the subjective judgments of high-level miliformation, and the relative importance of the several tary personnel into a numerical scale which is used categories of information, will vary from sector to in determining these parameters. This procedure sector. For example, an enemy airfield on the front also contains built-in statistical checks on the feasiline, where it is unusable, is relatively unimportant bility and validity of the model. 2 THE MODEL 2. 1 SECTORS AND CATEGORIES the equations without affecting the theoretical aspect of the model. It is to be hoped, for purely practiThe first step in building the model is to spec- cal reasons, that no great variety of situations ify the various types of sectors and categories of need be considered. information. The subdivision into sector types must first It is clear that the class to which a sector be made on an a priori basis. When equations have belongs is determined in part by depth (e. g., dis- been fitted to each sector type it may be observed tance from the front line), trafficability, and cover. that the parameters do not differ significantly However, interest in a given enemy sector also among certain of them, in which case these sector depends upon whether our mission is attack, hold- types may be identified. On the other hand, a poor ing, or withdrawal. In this first study only the fit of the equation for some sector type may indicate attack mission is considered. the need for a further breakdown into sub-types. So the model is set up with a preliminary categorizaAdditional distinctions might be introduced, tion into sector types with the expectation that modibut each new distinction greatly increases the labor fications will be made as experience with the model of experimentally determining the parameters of dictates. 2

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE Initially, depth is divided into four intervals; A displacement vector is defined by 0-3, 000 yds (battalion and regiment), 3, 000-15, 000 yds (division), 15, 000-40, 000 yds (corps), 40, 000 S = (s., s,..., ) (4) yds-200 mi (army and theater). Trafficability is divided into three degrees: (1) very good, as in j where s~ is the number of units of category i in open flat country with a road network; (2) medium; w and (3) very poor, as in marshy country. Similarly, sector j which are correctly identified but incorcover is divided into three degrees: (1) heavily rectly located. These four types of vectors are the variables in the model. wooded, (2) some cover, and (3) open and unprotected. Thus there are, potentially, 36 (4 X 3 X 3) distinct types of enemy sectors. However, it is The number of misses is now given by the vecanticipated that some of these distinctions will be unimportant and therefore some of the types can be Mj (mi 1... mi. m j (5) combined. Sectors are designated by superscripts 1''' n and the number of sectors is denoted by m. (Note that there may be many sectors of a given type.) where m1 = ti c - si 1 t 1 1 The items about which information is obtained are broken down into categories. Subscripts are Two further vectors serve as weighting facused to designate these categories and the number tors The first of these is of categories is denoted by n. These categories j j j include such items as artillery, tanks, infantry P p pi n)' (6) units, and airfields. Experience with the model will reveal whether some categories may be merged j and others need to be divided further. where pi measures the relative importance of knowledge of a unit of category i in sector j. The 2.2 THE BASIC EQUATION second is The general model is built around certain measurable integers which refer to information by cate- D (d. d,... d ), (7) =1.. 1.. n gory and sector. For a fixed sector j, t i de'notes the number of where di gives a categorywise evaluation of a disunits of category i (i = 1....... n) in this sector, play in terms of hits, false alarms, displacements, and the true situation vector in sector j is defined and misses. The equation for d and misses. The equation for d., to be 1 Tj (t t2, t t) (1) d = c f e + g s -i hi (8) 1 2 i n i 1 1 1 In a given intelligence display c! denotes the 1 was constructed after some preliminary investiganumber of units of category i in the sector j which tions discussed below, in which the displacement are correctly identified and located, and the co- term was not included. In Equation 8 the coeffirect identification vector in sector j is defined to be cients for the relaC= (c cJ, C... cJ,) (2) 1, 2, 1 n tive weights of hits, false alarms displacements, In this same display there may be some indi- and misses. The number d. is the net score for a cations of targets which in fact are not present. category of information in a given sector. Such an error is called a false alarm and a false alarm vector is defined by Equation 8 was normalized so as to make the l''*' ei.. e* *') coefficient of c! be 1. The effect of this is to choose j )~~~) 11 where e. is the number of false identifications of as the unit of measurement the value of a correct category i in sector j. identification of one unit of the given category. 3

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE This unit of measurement is of course arbitrary. The value or importance of information about a For example, the unit of measurement for infantry particular category of information in a particular might be in terms of platoons or companies, or any sector will depend on the command echelon evaluatother military unit. If for some displays com- ing the information. Information about a platoon of enemy infantry is of great concern to a front line panies were used as units in determining the d. and commander and of less concern to the divisional 1 commander and of less concern to the divisional for others platoons were used, it would be necessary commander. Conversely, the precise location of an j enemy airfield 100 mi in the rear is of much greater to determine a conversion factor to make the di from i value to higher command echelons than to lower one set of displays comparable with the d? from the ones. The importance vector PJ represents the 1 relative importance of the several categories of other. In the initial stages of constructing this information within a given sector from the point of linear model, the conversion factor may be chosen view of a command echelon which is concerned with a priori on the basis of relative manpower of the the welfare of all its subordinate units. respective units. However, it is reasonable that a particular surveillance device may yield information more valu2. 3 VALUATION INDICES able to one command echelon than to another and it A series of indices may now be constructed is therefore desirable to construct an index which representing evaluations of the outputs of a surveil- represents the relative values of an intelligence lance system or its components: display to different command echelons. The distinction lies primarily in the weighting of different sectors corresponding to their relative importance pi di (9) to different command echelons. The weighting vector is defined as is the net importance score for a category of information in sector j; m A(k) = a (k),..., a(k)..... a (k), (14) z pJ * DJ = _ p d (10) in which a command echelon is represented by k and i the sector designation by the superscript j. is the net importance score of all information in a The worth of a total intelligence display to a given sector; particular command echelon, k, is expressed by wj= P T (11) u(k) a(k) (15) is the worth of a sector; Analogues of Formulas 11, 12, and 13 would be needv ~- (12) ed in a complete development of this topic. In the wj present paper the treatment of echelon measures is limited to this brief introduction and no further use is the score on information in a given sector; and is made of Equations 14 and 15. These equations define a linear model-with the ~. zj vectors Tj, Cj, Ej, Sj objectively observed and v = (13) w with the vectors P and D to be empirically deter4j mined. Equations 8 and 10 are the basic equations. The parameters and range of applicability of these is the score for a theater. equations must be estimated from empirical data. 4

2144-152- T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE 3 EXPERIMENTAL AND MEASUREMENT PROBLEMS 3. 1 EXPERIMENTAL DESIGN to evaluate the importance of this situation. Hence, in determining the coefficients in Equation 8, some A number of special problems arise in applying the abstract model presented above. adjustment must be made which replaces d! by a In order to solve for the parameters of Equa- related quantity which represents what is really tion 8 it is first necessary to obtain numerical being judged. This substitution is discussed in Section 7. measures of the dj. There is, of course, a variety Because of the amount of labor required from of direct and indirect methods by which this might be done. The most direct method is to present a the military personnel by the Method of Paired be done. The most direct method is to present a Comparisons, a simplified variant, the Method of true situation and an estimate of it to qualified and Comparisons, a simplified variant, the Method of Rank Order, was adopted. In this method the overexperienced military personnel and to ask for a lay containing the estimate and the'map of the true numerical rating of the net score for that category lay containing the the map of the true of information in that sector. A highly indirect situation constitute the "object" to be judged and a of information in that sector. A highly indirect set of these is ranked in order from best to worst method would be to prepare detailed operational plans based on the estimate of the situation alone with respect to how well the estimate approximates and then have these operational plans evaluated inde the true situation. The worst display is given the pendently in the light of the true situation. In the number 0, the next worst 1, and so on. The numpretest to be described below, this method was ber thus assigned to the h-th display by the p-th explored and temporarily abandoned for practical officer is designated by bi (h, p) reasons. It is now being given further considera- 1 tion. On the other hand, the direct rating of estimates requires the assumption that a stable origin These rank orders are obtained from each of and unit of measurement exists which can be ap - a number of officers and averaged over the sample plied uniformly by all officers. Such a broad of officers for each display. The measure of the assumption does not seem justified here. Conse- value (designated b (h) of display number h is taken quently a relative method was used for evaluating 1 the estimates, rather than such an absolute method. as M This problem of obtaining a measure of the b. (h) (h, p) h=l, 2,..., N, (16) worth of a display of an estimate of a true situation 1 (N-1M for a given category of information (the di) is a 1 problem in psychological scaling. A method which in which N is the number of displays and M is the involves minimal assumptions and yet insures a number of officers used as judges. The range of j.! (h) is 0 to 1; a perfect estimate would be given solution for the di is to collect the data by the i Method of Paired Comparisons and to compute the the score 1. scale values by Case V of the Law of Comparative Judgmente This system is not completely satisfactory. For example, for M = 1 the difference between the In this comparative method of evaluating, given values assigned to any two objects is some multwo estimates each with its true situation, the basic tiple of regardless of how small the "psychojudgment requested from the officer is a decision N-1 about which estimate gives the better picture of its logical distance" between these objects may be. own true situation. Note that he is not asked also Obviously, for M small, Equation 16 will not adequately represent the relative values of two objects. Thurstone, L. L. "A Law of Comparative Judg- However, as M increases, it is reasonable to expect ment;" Psychol. Rev., 1927. No. 34, 273-286. that the percentage of judges who rank one object 5

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE ahead of another will be a function of the "psycho- this "object," which we may call A, as best, so logical distance" between them. That is, iftwoob- b(A) = 1. Now suppose B is another object which, jects, A and B, have a sufficiently small "psycho- ranked on an absolute scale, would rank as almost logical distance" between them, one could expect perfect. By the above method, since b(A) = 1, the number of judges ranking A ahead of B to ap- 1 proach M/2 as M becomes large. In this case, (B) N-1 -N' so that Equation 16 would distort ( 1-A) — b (B the "psychological distance" between objects A and B. However, this defect is easily remedied by becomes small. designing the experiment so that no estimate is perfect. In the present experiment one estimate of an Another defect with this method becomes evi- artillery situation was taken as perfect. In a rededent if an estimate of the corresponding situation is signed experiment no perfect estimates will be inperfect. Then, presumably, every judge will rank cluded. 4 THE PRETEST 4. 1 NATURE OF PRETEST With each of these true situation overlays was paired another overlay which was the estimat 3 of the true In order to test the general feasibility of t he opgeneral feasituation. Figure 4. 2-1 shows in red two examples erations necessary to determine the parameters in of intelligence estimates of enemy situations and in the model, a test model was prepared over a limited black the corresponding true enemy situations. range of categories of information within one situ- Each of such pairs will be referred to as a "stimuation. In preparation for this a pretest was run to lus"; more precisely, the "stimulus" may be retry out the materials, test the instructions, estimate garded as the psychological difference between the time limits, and obtain some preliminary notions of estimate and the true situation. The model is how well the model might work. The results of this designed to provide an objective basis for computpretest are given in Section 4. 5 and the more inten- ing this pyschological magnitude, the "distance" sive studies carried out at Fort Benning and Fort between the true situation and an estimate of it. Sill are discussed in Sections 5 and 6. There were eight stimuli prepared for terrain instances I and II respectively, and seven for terTwo categories of information were chosen for rain instance III. Within each terrain instance these this preliminary exploration: infantry companies stimuli were designed so that each component (hits, and artillery batteries, (i = 1 or 2), within a single false alarms, displacements, and misses) would sector, j = 1. cover a reasonable range. The table presented in Appendix C shows the composition of each stimulus. The situation was an attack mission on the part of our forces, the terrain had very good trafficability with moderate cover, and the sector was a regi- Eleven officers were used as judges in the premental front with a depth of 3000 yds. The actual test. Nine were Army officers ranging in grade from terrain chosen was in the neighborhood of Fort captain to colonel and varying in military experience Bragg, North Carolina, and three separate instances from exclusively technical and service to considerof the terrain were used: terrain instance I was the able combat command experience. One was a Navy Marston map; II, Silver Hill; III, Millstone Lake. captain and another was a colonel in the Marines. Although so varied a selection is not desirable for For each terrain instance a number of overlays the determining of the parameters of the model, it were prepared, each of which displayed a true served the purposes of the pretest quite well. Three enemy situation showing various dispositions of of the officers were used as judges in the experiinfantry companies only, of artillery batteries only, ment a second time after interval of two weeks or or of both infantry companies and artillery batteries. more. 6

'Pei Ul sl!un Awaua lo uoilr):)ol pur) jaqwnu lo sajowi4sa a:)ua6qj4uj -1:)r)lq ul s4!un I<wGuG 10 uO.I4r):)Ol Pur) u.lr)JJ9i SNoiivniis (mvwusg (INV i-vni:),v L-z-v oij............................................................................................................. 4. _:X,.....................................................................................................................................................................................................................................................................................................................:::::::::wwXw::-l::w x:::::: x....................................... w Equilibrium.............................. Approximation.......... w ie........... C i v il: w........................................................................................................................................:X..................................................................................................................................... w w.:. w w w..........!X:::: w................................:X:x............ Xx.......... X-V...................................................... w w:.............. X:::w w w aid,%....................................................................... Q........................................................................................................ -:X....................................................................................................:X: X X XX XXXXXV....:............................................................ F.-............................................................ w........... WX X:x ww-................................ w..........:w:.................................... w....................................... -..-..:.,::.. I . .. .. . ................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... x wwe. w. w 7lki(.................................................... w w............ w w w.............. X w........... w...................................................................... w w........................ w............................................................. X.......... ww..............:..................................... w............................................................................................................................................................................. w................................................. - W w w w w..................... 4.4 0, w w Jim!! w.......... iifilings jj jjj a cidic:............................................. w........... X w.... w

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE 4. 4 INSTRUCTIONS AND PROCEDURE approximated the true situation. Having completed these, he did the same thing for terrain instance II, The officer was first informed of the purpose ofd the for terrain instance and then for terrain instance III. Next these three the procedure. He was told that he would be given a ordered sets of stimuli were placed in three rows number of estimates of an enemy situation with their in front of him and he was required to pick sucrespective true situations and that he would be asked cessively the best one remaining in the three sets. to make comparative evaluations of how nearly the In this way all twenty-three stimuli were ranked. several estimates approximated their respective Each set was arranged in front of the officer so that true situations. he could see two successive stimuli in the set. Thus, he could easily reconsider and amend his ordering He was then given a copy of the terrain map and within each set as he proceeded. an overlay of one of the estimates, chosen at random, and was asked to consider generally what his Next, this procedure was followed in detail with operational plans would be to attack and achieve the ~~~~~~obj~~the stivemuli showing artillhe overlay. He waunits alone and wscreouwitofinally objective shown on the overlay. He was informed was carried out with the stimuli showing both artilthat he was a regimental commanding officer, that lery and infantry units. his capabilities and forces were reasonably adequate to accomplish his missions, and that he could, of 4. 5 RESULTS course, request additional support from higher headquarters. For each judge a rank ordering of the twentythree stimuli in each of the three sets (infantry After he had familiarized himself with the ter- alone, artillery alone, combined infantry-artillery) rain, the officer was presented the set of eight was obtained. Kendall's r (App. B), a rank correlastimuli for terrain instance I, with infantry units tion coefficient, was obtained among judges for each only, and asked to rank order them from best to set. These are presented in Tables 4-1, 4-2, and worst. The best was that for which the estimate 4-3. Table 4-1 presents the T coefficient among most nearly approximated the true situation and the judges for infantry alone, Table 4-2 for artillery worst was that in which the estimate most poorly alone, and Table 4-3 for combined infantry-artillery TABLE 4-1 INFANTRY DISPLAY ONLY Officer 1 2 3 4 1* 5 6 2* 3* 7 8 9 10 I 2.636 3.644.636 4.660.612.660 1.715.573.810.565 5.723.668.747.660.668 6.700.692.787.715.771.676 *2.747.739.636.542.644.660.621 *3.581.526.668.787.573.589.739.470 7.708.621.684.723.636.628. 731.628.715 8.589.549.676.771.581.708.628.447.644.597 9.763.613.597.636.644.597.573.668.534.692.526 10.739.668.573.486.557.621.581.779.383.526.487.581 11.621.660.455.478.439.613.470.700.312.526.462.573.692 *Officers 1, 2, and 3 served as judges twice. 8

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE TABLE 4-2 ARTILLERY DISPLAYS ONLY Officer 1 2 3 4 1* 5 6 2* 3* 7 8 9 10 1 2.708 3.510.502 4.518.581.407 *1.478.644.439.494 5.146.320.225.518.209 6.138.296.217.589.534.708 *2.700.731.399.407.549.099.138 *3.621.542.810.455.423.162.186.455 7.542.565.652.613.462.368.407.447.715 8.202.320.502.455.415.407.526.186.415.564 9.597.423.565.415.249.154.083.391.628.455.225 10.368.542.241.708.534.573.628.336.257.462.415.186 11.613.549.549.478.415.360.273.470.676.613.360.605.407 qOfficers 1, 2, and 3 served as judges twice. TABLE 4-3 INFANTRY-ARTILLERY COMBINED Officer 1 2 3 4 1* 5 6 2* 3* 7 8 9 10 1 2.312 3.399.375 4.375. 320.581 *1.470.494.605.510 5.320.621.470.510.510 6.209.415.581.399.605.415 *2.605.534.494.439.518.486.312 *3.462.304.597.534.644.455.391.431 7.431.407.494.660.549.565.415.573.447 8.170.289.534.518.399.320.352.312.344.510 9.542.107.273.407.344.217.123.368.209.399.344 10.526.581.462.360.518.549.375.478.431.415.202.304 11.700.494.455.304.415.478.273.636.328.391.225.502.660 Officers 1, 2, and 3 served as judges twice. 9

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE displays. The standard error* of these t's is the first seven officers were used in this analysis. of the order of 0. 3, so the probability that r will The officers whose judgments were utilized were: 1, exceed 0. 588 by chance is 0. 05 and the probability 2, 3, 4, 5, and 6, and a second run on 1. So, the that'r will exceed 0. 744 by chance is 0. 01. measure of the value of a display b. was based on the Three officers were used a second time as rankings of this sample of officers judges after-an interval of at least two weeks. The Kendall's W coefficients was used as a measure 7 coefficients among the three officers as compared of the conformity of agreement within the sample of with the - coefficient between the two repetitions of each of the three officers shows that the officers on average rank order correlation, pa average. the average agreed with themselves little more than average rank order correlation, p average. they agreed with each other, whether for infantry TABLE 4-4 stimuli, artillery stimuli, or the combined infantryartillery stimuli. Nevertheless, approximate test Community of Agreement in Ranking of significance indicates that the differences are far from significant. However, it should be recalled Pav that r coefficients have rather large standard errors. Infantry 0. 86 0.83 Because of the intermittent character of the Artillery 0.63 0.57 data collection in this pretest, only the judgments of Infantry-Artillery 0. 67 0.61 5 FORT BENNING STUDY 5. 1 NATURE OF FORT BENNING STUDY combat command experience ranged from a minimum of 3 months up to 40 months with an average of After the pretest but before all the data had 12. 6 months. (See Appendix F for copy of questionbeen analyzed, some revisions were made in the nare.) They were also asked about their staff stimulus materials and in the instructions, and experience, particularly in Intelligence (G-2) or arrangements were made with the Infantry School at Operations (G-3) sections. Seven officers had no Fort Benning for a number of officers with combat experience in G-2 or G-3 sections and one had 40 experience in World War II and in the Korean War months. The mean was 6 months. Most of their to act as judges. experience had been at the company and battalion level but, being field grade officers in the regular 5.2 TEST MATERIALS army, they were now orientated toward regimental Some slight changes were made in the number levels. However, their lack of regimental comof hits, false alarms, and misses on a few of the mand experience raises a question about whether stimuli. The values used are shown in Appendix C. their judgments should be exclusively relied upon for However, a considerable change was made in the determining the parameters of the equations. test materials themselves. A number of maps of each terrain were secured and the true situation for 5.4 INSTRUCTIONS AND PROCEDURE each stimulus was put directly on a ma. Then the After the experience with the pretest, a detailed estimate of the situation was printed or[ an overlay and uniform set of instructions and procedures were and the pair were always kept together to constitute prepared for the Fort Benning study. The officers the stimulus for judgment. were scheduled for the test two at a time at intervals of two hours at 8 and 10 AM and 1 and 3 PM. There 5.3 SUBJECTS were two complete sets of test materials and the The judgments of 19 officers at Fort Benning entire procedure rarely took more than four hours were secured. There were four lieutenant colonels, for any one officer. fourteen majors, and one captain. In a post-test interview with each officer it was learned that their Appendix B. 10

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE The two officers beginning the test procedure at TABLE 5-1 a particular time were first given a briefing covering the instructions and describing the procedure. AGREEMENT OF EACH JUDGE'S RANK ORDER This briefing was read to them in a casual and in- WITH THE AVERAGE RANK ORDER OF THE formal manner with discussion of questions if they REMAINING JUDGES arose. The prepared instructions are included as Appendix E. Officer Infantry Artillery Combined 5.5 AGREEMENT BETWEEN JUDGES 1 0. 874 0. 708 0. 648 The analysis of the Fort Benning data follows 2 0. 874 0. 621 0. 688 the same pattern as that of the pretest with some modifications in detail. As there were 19 judges, 3 0.842 0.731 0.672 the conformity of agreement among the judges was 4 0. 818 0. 628 0. 731 measured by computing the T coefficient between the rank order of the stimuli by each judge and the average of the rank orders by the other judges. 6 0. 802 0. 623 0. 632 This was done separately for infantry, artillery, and for the combined infantry-artillery. These 0. 794 0. 668 coefficients are reported in Table 5-1. 8 0. 791 0. 605 0. 731 To see whether the agreement of an officer with 0. 787 0. 7Z7 0. 64 the average of the other judges was specific as 10 0. 779 0. 648 0. 573 regards whether infantry or artillery units were being judged, the rank order of the officers on 1 1 0. 771 0. 771 infantry was correlated with their rank order on 12 0. 763 0. 644 0. 522 artillery. The 7 coefficient was 0. 105, not significant, indicating that the degree of agreement of an 0. 76 officer's rank order of the infantry stimuli with the 14 0. 743 0. 755 0. 557 consensus of the remaining officers is not related to his degree of agreement on artillery stimuli. The corresponding T coefficients between infantry on 16 0. 708 0. 692 0. 755 the one hand and artillery on the other with combined infantry-artillery stimuli were 0. 462 and 0. 129 respectively. 18 0. 688 0. 478 0. 549 The W coefficient for all 19 judges together was 19 0.636 0.415 0. 72 computed on infantry, artillery, and infantryartillery combined. The results, presented in Table 5-2, indicate a significant degree of agree- TABLE 5-2 ment. COMMUNITY OF AGREEMENT IN RANKING None of these results provided any secure basis W Pav for eliminating any of the officers from the sample; Infantry 0. 85 0. 84 however, the considerably lower correlations of the last three officers (17, 18, and 19 in Table 5-1) suggests the possibility that eliminating the data Infantry- 0. 65 0. 63 from them would improve the validity of the results. Artillery 11

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE 6 FORT SILL STUDY 6. 1 REASON FOR FORT SILL STUDY 6.4 ANALYSIS OF JUDGMENT CRITERIA IN In the analysis of the data obtained at Fort ARTILLERY SITUATIONS Benning, it was observed that the agreement among Figure 6. 1-1, 6. 1-2, and 6. 1-3 show that the the officers' judgments on situations involving in- two groups of officers agreed very well in the infantry alone was very good while the agreement on fantry situations and on the combined infantrysituations involving artillery alone was poor (Fig. artillery situations, but on situations containing 6. 1-1 and 6. 1-2). Since Fort Benning is an Infan- artillery alone, agreement was not very good. This try School, it seemed possible that a bias was re- tendency was also evident within each group. Acflected in the judgments of its officers. Accordingly, cordingly, an attempt was made to discover the it was decided to run a separate experiment at the Artillery School at Fort Sill and compare the results TABLE 6-1 with those from Fort Benning. The same maps and overlays were used as at Fort Benning and the same AGREEMENT OF EACH JUDGE'S RANK ORDER procedure was followed. WITH THE AVERAGE RANK ORDER OF THE REMAINING JUDGES. 6.2 SUBJECTS The judgments of eighteen officers at Fort Sill were secured. The average rank and staff experience of the officers involved at Fort Sill was some- Officer Infantry Artillery Combined what higher than that of the officers tested at Fort 1 0.917 0.751 0.743 Benning. There were two colonels, fifteen lieutenant colonels, and one major in the group. The aver- 2 0. 874 0. 490 0. 755 age combat experience was 8. 5 months; the average 3 0.866 0. 470 0.688 staff experience was 3. 5 months; and the average noncombat staff experience was 13 months. Much of 4 0. 862 0. 696 0. 743 the experience was at battalion and regimental levels. 5 0.842 0. 731 0. 775 6 0. 842 0.723 0.739 6.3 AGREEMENT AMONG JUDGES 7 0.818 0.743 0.791 As in the analysis of the data from Fort Benning, the agreement among judges was measured by com- 8 0.806 0.826 0.692 puting the Tr coefficient between the rank order of the simuli for each judge and the average rank order for the other judges. The results for infantry, ar- 10 0. 791 0.621 0. 751 tillery, and combined infantry-artillery are reported in Table 6-1. Also, the - coefficient of the officers' agreement among themselves for infantry and the 12 0. 767 0.688 0.660 officers' agreement among themselves for artillery 13 0.751 0. 652 0.688 was computed. The value obtained was -0. 196, indicating that the degree of agreement of an officers' 14 0. 743 0. 751 0. 739 rank order of the infantry stimuli with the average 15 0. 73 0.771 0.676 rank order is not related to the degree of his agreement on artillery stimuli. The corresponding r co- 16 0. 719 0.692 0.680 efficient between infantry and combined infantryartillery stimuli was 0. 51. The W coefficient for.. all eighteen judges together is presented in Table 6-2. 18 0.623 0.735 0.601 12

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE TABLE 6-2 TABLE 6-5 COMMUNITY OF AGREEMENT IN RANKING FORT SILL FORT SILL CCORRELATIONS OF INFANTRY-ONLY WITH | w l Pay INFANTRY-ARTILLERY COMBINED Infantry 0.87 0.88 F. B. Inf-Arty F. S. Inf-Arty Artillery | 0. 73 0.71 | F. B. Inf. 0. 822 0. 794 Infantry- 0.71 0.68 F. S. Inf. 0. 838 0. 783 Artillery TABLE 6- 3 RANKINGS OF SELECTED ARTILLERY SITUATIONS BY SUBGROUPS t c e s m (ec m) F.B. F.S. F.S. 3 3 0 0 0 1 1 1 3 3 0 1 0 3.4 2.5 2.4 2.0 2 2 0 1 0 5.7 4.4 3.6 3.4 3 3 0 2 0 10.4 6.1 7 4 Over -estimates 1 1 0 1 0 10.5 7.6 8.6 5.1 only only 2 2 0 2 0 12.5 9.4 9.4 6.1 3 3 0 3 0 17 11.3 12.2 8.1 1 1 0 2 0 17.4 16.2 19.6 9.9 2 2 0 3 0 20.5 15.8 14.8 8. 75 3 2 1 0 0 2.9 4.4 5.2 8.9 l Displacements 2 1 1 0 0 3.4 4.8 5.4 13.4 s only 3 1 2 0 0 5.3 14.7 14.2 19. 9 3 2 0 0 1 9.1 7.7 5.4 11.9 2 1 0 0 1 12.9 14 11.6 14. 3only {>Misses only 4 2 0 0 2 17.9 14.9 16 18.1 3 1 0 0 2 20.7 20.4 20.6 21. 3 TABLE 6-4 AVERAGE EXPERIENCE (Mo.) Reg. C. O. Bn C. O. Company Group Subgroup G2, G3 S2' S3' S4 or Exec. or Exec. C. O. F. B. I 8.3 9.7 0 4.0 II 3 * 9.6 F. S. I 4 5 0 3.0 II 6 0 9.4 3.25 4Only one man in the subgroup had this experience. 4Two men had this experience. 13

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE factors upon which the judgments in artillery situa- then the values of P. were found to be tions were based. The factors mentioned most J often in the debriefing reports were: P = 0. 737 and (20) (a) accuracy of reported strength and P = 0. 263, (b) accuracy of reported locations. where P1 is the weight assigned to the average rankSome officers stressed the first of these criteria and ings of the infantry situations and P2 is the weight others stressed the second. By grouping those who assigned to the average rankings of the artillery seemed to prefer one criterion over the other, both situations. Clearly, the infantry dominates the valFort Sill group and the Fort Benning group were uation put upon the estimate in the approximate ratio divided into subgroups. Subgroup 1 stressed accu- of three to one. racy of strength while Subgroup 2 stressed accuracy of location. Table 6-3 shows how each group ranked It seemed possible that the subgroups discussed typical situations. As may be seen in this table, in Section 6. 4 might differ materially in the weights F. B. 2 and F. S. 1 agreed very well. However, this they assigned to infantry and artillery. However, close agreement may not be very significant since upon investigation this was found not to be the case. the attitude of the Fort Benning group might be considered just a displacement of the attitude of the Fort Sill group. Table 6-4 shows the average expe- Slope=l rience (G2, G3, S2, S3, etc. ) in each subgroup. 1.0 6. 5 RELATIVE WEIGHTS- IMPLICITLY ASSIGNED BY JUDGES.8 In considering the rankings of the combined infantry-artillery situations, it was noticed that the correlation of the rankings of the infantry-only sit- / uations with those of the combined infantry-artillery,.6 situations was rather high. Table 6-5 shows the - correlations. A least squares procedure was used to determine the relative weights assigned to the infantry and artillery situation by the judges. If I< represents the average rank of infantry situation a, and.2 if A., and IA., are defined analogously to be the average ranks of artillery situation a and the combined infantry-artillery situation a, respectively, then fA = a I + a A and a = 1, 2,..., N. (17) 0.2.4.6.8 1.0 a 1 a 2 a b(h) F.S. The values of a and a2 were computed by minimizing 7( b (h)F. B b (h)F.S) = 095 C (/A -IA )2 (18) ca fa a FIG 6.1-1 PLOT OF AVERAGE RANK OF INFANTRY f a. SITUATIONS FROM FT. BENNING WITH P. = Ja and j = 1, 2, (19) THOSE FROM FT. SILL J a + a2 14

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE Slope =1 Slope 1 1.0 1 l.0.8.8.6.6,L.0;.,.@.41 d i cacla.4..2 _.2 0.2.4.6.8 1.0 0.2.4.6.8 1.0 b (h) F.S. b (h)F.S. ( -h)F -B, (h) F.S.) 071 - ('(h) F.S. b(h)FB 0.85 FIG 6.1-2 PLOT OF AVERAGE RANK OF ARTILLERY FIG 6.1-3 PLOT OF AVERAGE RANK OF INFANTRYSITUATIONS FROM FT. BENNING WITH ARTILLERY SITUATIONS FROM FT. BENNING WITH THOSE FROM FT. SILL THOSE FROM FT. SILL 7 DEVELOPMENT OF THE MODEL 7. 1 GENERAL CONSIDERATIONS ual threshold. (In this section subscripts and superscripts are omitted..) The purpose of this study is to devise a linear measure on an absolute scale of the merit of an in- In our problem, c, e, and m are fixed in advance; telligence estimate of an enemy situation. The idea d is calculated experimentally, and the constants f of using a score of the form d is calculated experimentally, and the constant s f of using a score of the form and h are to be determined by linear regression so d = c - fe - hm, (21) as to give a least squares fit. where c, e, and m are, respectively, hits, false In early analyses, Formula 21 was used. Howalarms, and misses, and f and h are constants, was ever, after a more careful examination of the data suggested by some experimental studies in vision and after more information was obtained in the deresearch. In that application, values for f and h briefing, some modifications were introduced. First, were assigned by the experimenter, and one purpose it became clear that the officers almost without exof the experiment was to see how changes in f and h ception regarded as a displacement what had been would influence the subject in responses near the vis- intended as a miss plus a false alarm. To handle 15

2144-152- T THE UNIVERSITY OF MIC-HIGAN ENGINEERING RESEARCH INSTITUTE this, the concept of displacement was introduced and in which s, e, and m are defined by Equations 22, 23, the following definitions were made: and 24 and f, g, and h are constants to be calculated by linear regression from the observed d values. If the true situation has t units and the estimate Perhaps further modifications of Equation 25, which shows t units, then c is defined to be the number of would include s as one of the variables, should be units whose positions on the two maps coincide, considered. Next, _ 7.2 TRANSFORMATIONS OF EMPIRICAL VALUES s = min (t - c, t - c) (22) TO VALUES OF d is defined to be the number of displacements, In the foregoing discussion reference has been e = t - c - s (23) made several times to "observed" or "experimentally determined" values dof d. These terms are is defined to be the number of false alarms (errors), andnot strictly correct. The experimental results yield directly only each officer's rank order of a set of m = t - c - s (24) maps and an average, normalized score b = b (h), is defined to be the number of misses. Equation 16, which is assumed to be related to d = d(h). Also, s is defined as the number of near misses (i. e., displacements such that the distances between In essence, the officers based the rank of a the true units and the corresponding units on the given estimate of the corresponding true situation estimate are less than some fixed amount, e. g., 300 upon the effect the errors in the estimate might have yds for infantry). These quantities are illustrated on an attack and not on the value of t in the true sitin Figure 7. 1-1 Targets 1, 2, and 3 coincide with uation. Thus, two perfect estimates (t = c, s = m = Units 6, 7, and 8 of the estimate and Element 10 is e = 0) would be judged equally good even though one a near miss for Target 4. had a much larger value of t. This suggests that b should be matched with -, i. e., assuming - to be Next, three methods of using Equation 21 were t t considered, each of which depends on replacing c, a linear function of b and fitting by linear regression. e, and m by combinations of c, e, s, g, and m as There are some reasons for using dor defined above. These methods (exact, judged, and d (t + e) (t+ e + s) strength) are defined in Table 7-1. instead of -. For example, t + e + s is the total TABLE 7-1 | number of objects seen when the estimate is overTHREE POSSIBLE CRITERIA FOR laid on the true situation and thus might have some MEASURING THE VARIABLES psychological meaning. Inclusion of e, or some c e m multiple of e, in the denominator bounds the range Exact c e + s m + s of the fraction. of the fraction. Judged c+s e+s- s m+s- It is very difficult to decide a priori which of the many possible forms is the best. If b is matched Table 7-2 shows the values obtained by applying with d/a where a may be t, t + e, or some similar these criteria to the situations in Figure 7. 1-1. expression, and values of f, g, and h are found by TABLE 7-2 linear regression theory sometimes some possibilMEASURES OBTAINED FROM ONE SITUATION ities can be eliminated. For example, if, when b APPLYING THREE DIFFERENT CRITERIA is matched with d/a, one of the coefficients in Equac e m tion 25 is less than zero, this form is unsatisfacExact 3 3 z2 | tory. Similarly, f>l or g>l is, on the face of it, Judged 4 2 1 illogical, for g->l means that a displacement is better than a hit, which is absurd, and f > 1 is unlikely, L Strength 5 1 0 ] judging from the data from Fort Sill and Fort Of these criteria some modification of the "judged Benning. If b is matched with d/a, gave the best fit, but it was apparent that even better results might be obtained by replacing Equation 21 c e s m with d =c -fe + gs-hm, (25) b a ( 2 16

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE By the procedures described in Appendix A, values b(h) with b(h) was obtained where of f, g, and h were obtained for a = t, t + e, and c s + m( (h) = (31) t + e + s. Values were also obtained for a special t + e t + ee t + e case'c fe gs hm and b = c (27) c4L = c, t t + ae t t S*= 0.3s -0.3s' - 0.6s' -0. 75s" + 0.4(s" + s"), for a range of values of a. As noted in Appendix A, 2 3 1 2 3 linear regression theory does not apply to either the e* 1. 25e' + 0.8e" + s' + s' and latter case or to the case of a = t without some re- 2 3 striction. mA = m' + 0. 4m". Here Tables 7-3 and 7-4 give the values of f, g, and s' = the number of displacements < 300 yds on h obtained from the Fort Sill data using the above the front line, cases and present the correlation of the data from Fort Benning with the corresponding b's. s' = the number of displacements between 300 ydE and 600 yds on the front line, Another approach is to redefine s, the displacement term. The distance between a reported tar- the front line, get and the true target can be denoted h. Then the number of stimuli for which m' = the number of front line misses, and Si = hi - i( h 4 hi (28) e' = the number of front line false alarms. Double primes indicate the same quantities in the where ho = 0 and hi is chosen arbitrarily for i >,1 rear areas. (Note: A front line position with poor can be defined. Then, for i =. 2, and 3 trafficability is counted as a rear position.) s = si + sZ + s3 (29) Figure 7. 2-1 shows the correlation between the and the basic equation becomes values of b(h) obtained in this way and the observed (h) c- fe values, b(h). The values of the coefficients of si, c(h) f -s + g2 S + g 3S + 9 S hm, (30) - a1 e 1 g2 s2 g3 s3 s and the rest were chosen arbitrarily to yield a oa ca a good correlation and no attempt has been made to where a may be chosen as in the preceding section. develop a theory for optimal choice of these coeffiIn the course of examining this method it was noted cients. that the judges weighted front line misses, displacements, and false alarms differently from corre- The results indicate (1) the need for splitting sponding errors in the rear area. Letting x' denote sectors so as to consider front line and rear areas a variable on the front line and x" denote the same as separate sectors and (2) the possibility of a good variable in the rear area, a high correlation of fit by simple nonlinear formulas. TABLE 7-3 THE COEFFICIENTS AND FIT OF EQUATION 26 FOR THREE VALUES OF a Infantry Case f g h,bF.S.) AbF.B.) a = t 0. 149 0. 701 0. 240 0. 735 0. 751 a = t + e 0.073 0. 280 0. 474 0.806 0. 814 a = t+ e + s 0. 137 0. 646 0. 742 0. 775 0. 783 Artillery Case f g h r(b, bF. S.) r(b, bF. B.) a =t 0. 274 -0.061 0. 298 0.830 0.644 a = t + e 0.098 0.145 0.381 0.719 0.830 a = t + e + s 0.148 0.297 0.248 0.858 0.640 17

2144-152 - T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE TABLE 7-4 THE COEFFICIENTS AND FIT OF EQUATION 27 FOR FOUR VALUES OF a Infantry Case f g h 1 (b, bF. S.) (b, bF. B.) a = 0 0. 149 0. 701 0. 240 0. 735 0. 751 = 0.5 0. 593 0.617 0.545 0. 787 0. 779 a = 0.75 0.814 0.631 0.576 0.771 0.751 a = 1.0 0.972 0.620 0. 585 0.775 0. 743 Artillery f g h T(b, bF. S. ) -(b, bF. B. ) a = 0 0. 274 -0.061 0. 298 0. 830 0. 644 a = 0.5 0. 579 -0.016 0.352 0. 838 0. 668 a = 0.75 0. 717 0.003 0.367 0. 862 0.684 a = 1.0 0. 820 0.016 0. 363 0. 802 0. 680 1 Objective Objective - \ ~~~~18~ 5) True Situation I Intelligence Estimate 18

2144-152- T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE 1.0 0.8 o Ft. Benning 0 * Ft. Sill o > 0.6 t - COMPUTED VALUES 8 UI) O0 0 ~ r(b, b) =0.87 0.2 c* 0 0 0.2 0.4 0.6 0.8 1.0 b - COMPUTED VALUES FIG. 7.2-1 COMPARISON OF OBSERVED VALUES WITH PREDICTED VALUES (bnfantry Situations Only) 8 CONCLUDING REMARKS 8.1 SUITABLE MODEL One would expect that, if two stimuli are nearly It seems clear that a suitamongdel can be equal in value, some judges would prefer one and some the other and that the amount of overlap could built although its final form can be developed onlyof (A) The method of collecting data gives individual aftnk order study. Several of the forms already use t ge ance min (-hm, -fe) em. (33) tdisplays). Tthe awdtveryg wel ank order cor t u se of the average rank order a s an empir bound of the was takenl a numerical measure obande- trdvl of.ea display should be the total numeber tween predicted and so as to prvaluesent a representa- of units present. However, the linearity in e is estimates. These correlations are almost as high as is possible in view of the degree of agreement (B) The basic scoring equation among the judges.ts the desirability of replacing e in d = c - fe + gs - hm (32) gives for d a theoretical range of (A) The method of collecting data gives individual rank orders of the stimuli (in this case intelligence mi (-hm, -fe) < d < t. (33) displays). The average rank order of each stimulus It seems reasonable that the upper bound of the 19

2144-152 -T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE Formula 32 by some function of e and of t which is a = t the coefficients of the parameters are dependbounded in e. For example, one proposal is ent. Some independent method for computing the mean of the 6 values would eliminate this difficulty. em t) 4 Further analysis of this problem is indicated. 8.3 FUTURE DEVELOPMENTS or, more generally, In addition to work on the unsolved problems A, e(t + ke) k. B, C, and D the following investigations are planned: (E) There seem to be two major points of view A still more general possibility is represented in our artillery data. Officers in one e a F(t, e), (34) group were interested primarily in pinpoint location of enemy artillery so as to direct counterfire. where F is a non-negative function for which Officers in another group were interested primarily in accurate knowledge of the number of opposing lim F(t, e) = 1 (35) batteries. Clearly both points of view are important. t- o However, the data are insufficient to give appropriate weights to these two factors. There are two possible approaches to this problem: (a) use higher e-lim eF(t, e) exists and is finite for all t ranking officers as judges, or (b) redesign the experiment. The purpose in (a) is to obtain judgThe crux here is optimum selection of the function ments from officers who have been in command F. simultaneously, of infantry troops, and of artillery. The most probable course is to combine (a) and (b). (C) As was noted earlier in this memorandum, the judges do not make their preference choices on (F) Analysis of data already on hand should be the basis of the d values but rather on b = d in continued. This involves, in particular, close study which a is t or t + s, or t + e + s or some similar of the data on combined infantry and artillery disquantity. Good fits to the data can be obtained with plays. any of these. Further study, both theoretical and experimental, is necessary to discover the true (G) A new set of displays should be designed to nature of a. include at least three types of information and a larger front sector. Preparation of this new material (D) In the special case a = t, a mathematical will, of course, build on what has already been difficulty is encountered in obtaining a least squares learned and will require continuing close touch with fit to the data. The observed values b are matched the military liaison. This material would be inagainst the computed value f, given by tended for judgments by officers of somewhat higher rank than were those in the earlier experiments. = xd + y. The value of the debriefing records from the experiments of last summer indicates the desirability of As is shown in Appendix A, Equation 45, when paying even more attention to this feature. 20

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE APPENDIX A METHODS OF DETERMINING COEFFICIENTS N equations represented by Equation 41 are in a FOR BASIC EQUATION form suited to the least squares procedure. Each A estimated value bi corresponds with an observed Indexes referring to sector or category will be value 1i associated with a particular display. It is omitted in this section. The subscript employed desired to minimize the variance of errors of prewill denote a given display. diction, an error of prediction being Equation 26 of Section 7 may be written in the Pi b - bi (42) form Summing over the displays provides 3 = c' - fe' + gs! - hm! and i = 1, e....,N, where i i [ +i+a3si +a4m+5i=l i=l (43) Xi =Xi (39) 1 a The normal equations in a1, a2,, a3, a4, and a5 a may be taken as 1, t, t+e, t+e+s, or some similar expression, and which minimize p i are obtained by requiring m. = (t. - c. - s.).. (40) Equation 38 is not in the best form for appli- a = and i = 1 2, 3, 4, 5. (44) cation of linear regression procedures. In order to 1 adjust the mean and the scale of the predicted values This yields five equations which are linear in al, to those of the observed values, Equation 38 is a, a3, a4, and a5. For = t + e and a = t + e + s, replaced by a linear transformation of it: the coefficient matrix of these equations is nonb a + e singular, which allows solving for the unknown. ib i 2 i 3 i 4 (41) However, for a = t and the special case Then c1 e S m A 1 i i bi = a - + a + a - + a + a (45) 1 1t 2 t.+- eye. +3 t. 4 t. 5 d ^ 1 1 1 1 1 is an estimate of d., al 1 the coefficient matrix is singular. If we require a5 = 0 and use only the first four equations of the set, _a2 the reduced coefficient matrix is now nonsingular a is an estimate of, and we may solve for al, a2, a3, and a4. Setting a1 a5 = 0 has the disadvantage that it forces an artifia cial mean for values of bi obtained and results in a 3is an estimate of g, and distortion of the values of f, g, and h. Although 1a this is undesirable, it seemed worthwhile to proceed, since the values of a5 in the cases a = t + e and a4 a = t + e +s were relatively small. However, this a is an estimate of h. method is not considered adequate and further cona1 sideration will be given to developing a more satisThe values of c', e', s!, and ml are known and the factory method. 1 1 1 1 21

2144-152- 1T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE APPENDIX B DEFINITION OF r, W, and Pav The standard error of -ris of the order of. In the above example, r = 0. 333. In order to measure the conformity ot agreeIf the degree of agreement between two rankings ment between M observers in ranking N objects, it of N things is to be computed, the Kendall r is a is necessary to define simple and easily computed measure. r is com- is necessary to define puted as follows: Three things x1, x2, and x3 are ranked according to some property by two observers S (d ) = - M(N+ 1) - g(h, p), (50) A and B, thus: h=l 1p =l Object A B in which g(h, p) is the rank assigned to the hth x1 1 2 object by the pth observer. Then x2 2 1 ~ W = 12 l S ~and 0, W, 1 (51) 3 3 3 = M (N3 _N) If the ranking by A is considered the natural order, is a measure of the conformity of agreement among then the observers and 1 S =.-N(N-1)- (twice the total number of inversions). MW- 1 2 * (46) Pav M- 1 Here S = 1. Then sHere S = 1. Then Sis the mean value of the Spearman coefficient among N(1 (M) N(N - 1) (47) the 2 possible pairs of observers. Since 1 1 To obtain the values used after the concept of - N(N - 1)$ S N(N - 1), (48) displacements was incorporated, terms were defined thus: therefore - 1~,,< 1 (49) s = min (el, t-c), and r = 0 corresponds with a random correlation. e = e - s, and For a more complete discussion of rank correlation and the quantitiesr, W, and Pav, see Rank Correla- m = t - c - s tion Methods, M. G. Kendall, Charles Griffin and Company, Ltd.; 1943, pp. 26 and 81. where e1 represents the values of e in Appendix C. 23

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE APPENDIX C COMPOSITION OF THE STIMULI Terrain Terrain Instance true c e t-c No. Instance true c e t-c No. 3 a 1 0 5 1 5 1 a 4 4 2 0 6 b 1 1 3 0 4 b 3 2 2 1 4 1 a 1 1 5 0 6 2 a 5 3 0 2 3 b 3 2 3 1 5 b 3 1 2 2 3 2 a 1 1 4 0 5 3 a 5 3 2 2 5 b 3 2 0 1* 2 b 3 3 2 0 5 1 a 2 1 4 1 5 3 a 5 5 1 0 6 b 3 3 0 0 3 b 2 2 2 0 4 2 a 2 2 4 0 6 1 a 5 3 3 2 6 b 2 1 3 1 4 b 2 1 2 1 3 1 a 3 2 1 1 3 3 a 6 3 3 3 6 b 2 2 3 0 5 b 3 2 1 1 3 3 a 3 3 2 0 5 3 a 6 3 0 3 3 b 3 3 3 0 6 b 3 1 0 2 1 2 a 3 3 3 0 6 2 a 6 4 1 2 5 b 2 2 0 1 2 b 3 1 3 2 4 2 a 4 2 3 2 5 1 a 6 4 2 2 6 b 2 2 1 0 3 b 3 1 1 2 2 3 a 4 2 4 2 6 1 a 6 5 0 1 5 b 2 1 1 1 2 b 1 1 1 0 2 1 a 4 3 2 1 5 2 a 6 5 1 1 6 b 2 1 0 1 1 b 1 1 2 0 3 2 a 4 3 3 1 6 b 3 3 1 0 4 This table gives the specifications for each of 23 displays of infantry alone, 23 displays of artillery alone, and 23 displays of combined infantry and artillery. The row headed (a) indicates infantry. The row headed (b) indicates artillery. No. Designates total number of units displayed. *This number is correct for artillery alone but on the combined infantry-artillery there are two misses of artillery positions. 25

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE APPENDIX D RANKINGS OF SITUATIONS BY INDIVIDUAL OFFICERS Table D- 1 Fort Benning Infantry Rank Orders Page 28 Table D-2 Fort Benning Artillery Rank Orders Page 28 Table D-3 Fort Benning Infantry-Artillery Rank Orders Page 29 Table D-4 Fort Sill Infantry Rank Orders Page 29 Table D-5 Fort Sill Artillery Rank Orders Page 30 Table D-6 Fort Sill Infantry-Artillery Rank Orders Page 30 27

2144-152-T THEUNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE TABLE D-1 FORT BENNING INFANTRY RANK ORDERS 10 bj 11 12 13 14 15 16 17 18 19 (h) i I-I 1 1 1 3 1 1 2 1 1 1 1 3 3 1 2 1 0 1 8 0. 079 1-2 3 3 3 2 5 2 3 2 4 3 2 6 2 3 9 2 4 4 9 0. 170 1-3 20 17 20 19 21 20 20 20 20 21 18 19 16 21 19 20 19 19 18 0.878 I-4 10 11 9 12 15 9 11 11 11 15 10 10 14 7 13 12 7 7 11 0.490 1-5 16 16 14 16 13 16 15 17 13 18 19 18 17 16 20 14 10 20 19 0.734 1-6 15 15 16 18 18 17 17 14 15 7 9 16 13 18 15 13 16 16 15 0.677 1-7 21 21 21 20 20 22 19 21 21 20 21 21 19 19 16 17 21 18 20 0.904 1-8 18 19 18 14 17 18 18 15 18 17 17 11 20 17 17 19 11 10 16 0.724 1-9 8 5 5 1 9 5 5 8 7 9 ll 1 0 10 11 16 1 3 6 0.289 1-10 2 2 2 4 2 4 1 3 2 2 3 4 4 2 3 3 3 11 1 0.139 1-11 6 7 13 9 7 3 8 13 10 13 7 13 9 6 6 6 12 5 4 0.376 1-12 7 6 12 5 8 12 6 10 14 8 5 7 8 ll 1 9 17 2 7 0.371 1-13 5 4 4 6 3 7 4 5 3 4 4 5 5 5 4 4 5 9 5 0.218 1-14 12 9 7 ll 10 8 14 6 6 6 13 8 7 8 l0 7 8 13 13 0.421 1-15 11 14 10 17 6 10 7 7 8 10 12 9 15 13 14 10 14 14 12 0.510!-16 22 22 22 22 22 21 22 22 22 22 22 22 22 22 22 22 22 21 22 0.995 1-17 4 12 6 7 4 6 13 4 5 5 14 2 10 15 7 8 6 8 3 0.333 1-18 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 2 0 0 0.007 i-19 14 13 15 15 16 13 16 16 12 12 15 14 18 14 18 11 13 17 17 0.667 1-20 9 8 11 8 12 14 10 12 16 14 6 15 6 9 5 18 15 6 2 0.469 1-21 13 10 8 10 11 11 9 9 9 11 8 12 11 4 8 5 9 15 14 0.447 1-22 19 20 19 21 19 19 21 19 19 19 20 20 21 20 21 21 20 22 21 0.911 i-23 17 18 17 13 14 15 12 18 17 16 16 17 12 12 12 15 18 12 10 0.672 TABLED-2 FORTBENNINGARTILLERYRANKORDER ~ ~ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 bj (h) O i A-1 9 6 3 12 1 4 7 5 3 5 2 9 7 0 5 9 1 6 4 0. 234 A-2 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 1.00 A-3 3 4 2 8 2 9 9 6 2 1 4 7 8 7 12 2 3 14 13 0.278 A-4 4 7 11 3 12 12 13 15 10 14 8 3 11 13 7 5 13 8 11 0.431 A-5 15 16 17 14 10 8 8 7 15 13 14 11 9 6 10 13 17 7 8 0.522 A-6 10 12 12 11 9 5 5 8 9 12 9 6 6 12 4 4 9 5 5 0.366 A-7 6 10 7 2 8 6 2 2 8 11 5 4 5 8 11 8 7 4 6 0.287 A-8 8 5 16 15 15 15 19 16 12 15 20 12 14 14 13 16 16 13 19 0.653 A-9 17 18 15 16 19 2 15 3 16 19 1 14 10 10 2 i0 20 2 0 0.500 A-10 2 2 8 1 11 10 12 14 7 6 3 5 12 4 8 12 0 1 3 0.289 A-11 18 17 19 18 17 19 18 17 18 17 19 20 18 20 20 17 15 20 20 0.830 A-12 13 13 10 10 16 3 4 1 14 10 6 13 4 3 1 6 10 3 1 0.337 A-13 7 8 5 4 5 1 3 4 6 4 7 8 3 2 6 3 4 9 7 0.230 A-14 11 15 13 6 13 18 14 18 17 16 13 18 16 16 19 15 12 10 17 0.663 A-15 21 19 21 19 21 21 20 19 19 21 21 21 21 21 21 21 18 21 21 0.919 A-16 5 3 6 5 6 11 6 10 4 9 12 2 1 11 9 1 6 16 14 0.328 A-17 0 0 0 0 4 7 0 9 1 3 0 0 2 1 0 7 8 0 2 0.105 A-18 1 1 1 7 0 0 1 0 0 0 10 1 0 5 3 0 2 15 10 0.136 A-19 14 9 4 9 3 13 10 11 5 2 11 16 13 9 14 11 5 18 16 0.462 A-20 16 14 9 17 14 20 17 13 13 8 16 19 17 15 17 18 14 19 18 0.703 A-21 19 21 20 21 18 16 16 21 20 20 17 15 19 19 18 20 19 12 12 0.821 A-22 12 11 14 13 7 14 11 12 11 7 15 10 15 17 15 14 11 17 15 0.577 A-23 20 20 18 20 20 17 21 20 21 21 18 17 20 18 16 19 21 11 9 0.831 28

2144-152-T THEUNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE TABLE D-3 FORTBENNING INFANTRY-ARTILLERYRANK ORDERS O 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 bj (h) i IA-1 1 1 1 2 1 1 1 1 1 1 6 1 1 0 5 2 0 7 1 0. 081 IA-2 5 9 4 7 11 12 5 8 6 13 11 18 9 6 11 7 9 12 6 0.404 IA-3 13 16 19 10 14 13 18 19 13 14 13 16 12 17 13 18 4 20 7 0.644 IA-4 4 14 5 9 17 19 11 10 12 18 14 3 14 8 4 10 16 6 11 0.490 IA-5 17 13 13 17 13 14 19 12 9 11 16 19 13 19 15 15 7 15 17 0.656 IA-6 16 17 11 18 12 9 16 15 14 12 15 11 19 5 8 12 18 9 5 0.579 IA-7 22 22 20 20 16 7 17 20 21 16 17 13 21 18 16 17 20 14 9 0.780 IA-8 15 18 10 11 18 22 10 14 19 20 18 7 20 10 12 22 13 13 21 0.701 IA-9 12 8 9 1 6 2 3 7 8 15 1 9 0 4 3 9 10 4 8 0.285 IA-10 7 5 2 6 2 4 2 2 3 4 2 6 4 7 9 1 3 8 14 0.218 IA-11 6 4 18 14 19 17 15 18!6 9 9 10 16 9 7 6 17 3 13 0.541 IA-12 11 2 14 3 15 5 8 6 ~5 5 3 2 11 2 0 8 21 10 2 0.342 IA-13 2 3 3 5 3 3 4 4 2 2 10 5 3 11 6 3 1 11 4 0.203 LA-14 8 7 7 13 7 11 14 5 5 10 12 12 15 16 18 11 12 16 12 0.505 IA-15 9 15 12 16 8 15 13 9 11 7 20 22 22 15 19 16 6 19 19 0.653!A-16 21 20 22 22 22 21 22 22 22 22 22 20 17 22 20 21 15 21 22 0.947 IA-17 3 11 6 4 5 8 6 3 4 3 4 4 5 13 2 4 8 2 3 0.234 iA-18 0 0 0 0 0 0 0 0 0 0 00 0 2 1 1 0 2 0 0 0.014 L&-19 10 12 17 12 9 6 12 13 7 6 7 17 8 20 21 13 5 17 18 0.550 iA-20 14 6 15 8 ~10 10 9 16 17 8 5 8 10 12 17 14 11 5 15 0.502 iA-21 18 10 8 15 4 18 7 11 10 17 8 14 6 3 10 5 19 18 10 0.505 IA-22 20 19 21 21 20 20 21 21 20 19 21 21 18 21 22 20 14 22 20 0.911 1_A-23 19 21 16 19 21 16 20 17 18 21 19 15 7 14 14 19 22 1 16 0.754 TABLED-4 FORTSILLINFANTRY RANK ORDERS o.~ O 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 b (h) I-I 1 1 1 1 1 1 1 1 1 1 1 0 1 0 2 3 5 2 0.061 1-2 2 2 5 7 6 3 2 0 2 4 9 2 8 3 6 6 7 4 0. 197 I-3 19 19 20 19 21 19 20 21 19 19 19 18 21 21 18 14 18 20 0.871 I-4 11 12 11 11 9 i1 12 11 10 13 15 13 13 11 11 0 8 10 0.510 1-5 17 18 15 14 17 15 18 15 13 14 14 17 11 14 19 13 16 15 0.694 I-6 14 16 17 17 10 17 16 17 14 17 17 15 15 16 17 16 15 16 0.712 I-7 20 20 18 20 20 20 21 20 21 21 20 20 19 20 21 21 20 21 0.912 1-8 18 17 21 15 11 14 15 18 17 18 18 19 17 18 13 19 17 18 0.765 1-9 10 5 8 3 5 10 4 8 7 6 2 8 7 5 1 0 1 7 0.245 1-10 4 3 2 9 2 2 3 2 3 2 3 3 3 2 4 2 13 3 0.164 1-11 9 13 6 6 7 5 8 10 12 11 11 12 14 17 9 7 9 11 0.447 1-12 3 9 9 2 8 12 10 12 8 8 5 10 6 9 7 5 2 6 0.331 1-13 5 4 4 10 3 4 6 4 5 3 4 4 2 4 5 9 12 5 0.235 1-14 7 8 10 13 13 8 9 5 9 5 8 9 9 7 8 11 14 8 0.407 1-15 13 11 13 8 16 7 13 9 16 9 7 7 10 12 10 15 10 13 0.503 1-16 22 22 19 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 0.992 1-17 8 6 14 5 4 6 5 6 4 10 6 11 12 8 3 8 3 0 0.301 1-18 0 0 0 0 0 0 0 3 0 0 0 1 0 1 0 1 0 1 0.018 1-19 16 14 16 20 15 18 14 13 15 7 13 16 16 13 15 18 19 14 0.687 1-20 12 10 7 4 12 13 11 14 6 15 12 5 5 10 12 4 4 9 0.417 1-21 6 7 3 16 14 9 7 7 11 12 10 6 4 6 14 12 6 12 0.409 1-22 21 21 22 21 19 21 19 19 20 20 21 21 20 19 20 20 21 19 0.919 1-23 15 15 12 12 18 16 17 16 18 16 16 16 18 15 16 17 11 17 0.705 29

2144-152-T THEUNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE TABLE D-5 FORTSILL ARTILLERY RANK ORDERS 1 2 3 4 5 6? 8 9 lo ll 12 13 14 15 16 17 18 ~ b~(h) A-1 7 11 8 5 2 9 8 12 5 5 8 9 8 5 10 5 6 10 0. 336 A-2 22 22 22 22 22 22 22 22 21 22 22 22 22 22 22 22 22 22 0.997 A-3 14 10 12 15 13 12 15 7 14 4 5 14 14 3 14 7 15 6 0.490 A-4 6 6 16 13 14 11 2 5 11 13 7 12 6 11 4 14 11 18 0.455 A-5 9 12 10 9 9 14 12 18 10 14 6 I1 5 14 0.495 A-6 8 9 7 6 7 3 7 11 7 7 9 8 3 9 8 8 7 9 0.336 A-7 1 5 6 3 3 6 1 2 4 4 10 1 2 2 8 0.174 A-8 17 15 17 19 19 10 17 8 17 15 19 18 16 15 19 16 18 20 0.745 A-9 4 8 9 1 1 7 5 20 6 6 4 14 5 4 1 0.293 A-10 2 7 4 4 8 8 4 2 1 9 13 5 10 7 2 9 4 5 0.2 A-11 20 19 20 20 20 20 19 16 22 20 20 19 19 18 20 20 20 21 0.891 A-12 3 4 3 0 0 5 3 17 3 1 1 0 0 12 6 3 0 7 0.172 A-13 5 2 5 8 6 2 6 4 4 0 3 1 7 4 7 0 5 4 0. A-14 11 16 18 14 15 18 10 15 19 19 14 17 12 17 3 12 15 0.667 A-15 21 21 21 21 21 21 21 19 20 21 21 21 19 21 21 21 16 0.929 A-16 16 3 1 17 11 14 10 15 2 15 2 16 11 14 3 0.422 A-17 0 1 2 2 5 0 0 1 0 8 0 3 1 1 0 1 3 0 0.071 A-18 13 0 0 7 4 4 11 0 12 2 12 7 13 0 13 6 13 2 0.301 A-19 15 13 11 10 16 13 16 10 15 11 17 13 17 6 15 12 17 1 0.576 A-20 19 ~7 15 18 18 16 20 14 18 16 18 20 20 16 18 15 19 11 0.778 ~-21 10 18 19 11 10 19 9 21 9 12 11 10 9 20 9 17 9 17 0.606 A-22 18 14 14 16 17 15 18 9 16 17 16 15 18 8 17 13 16 12 0.679 &-23 12 20 13 12 12 17 13 13 8 18 10 16 11 21 11 18 10 19 0.641 TABLED-6 FORTSILL INFANTRY-ARTILLERYRANK ORDERS O 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 (h) IA-1 2 1 1 1 5 2 2 1 1 2 0 1 2 0 5 5 8 1 0.101 IA-2 8 6 6 11 3 8 7 9 6 1 10 17 12 4 17 8 7 7 0.371 IA-3 20 13 10 17 20 13 16 6 18 19 14 16 16 20 19 14 20 14 0.720 IA-4 11 12 12 14 4 7 9 7 14 14 7 12 8 11 9 11 5 11 0.449 IA-5 13 16 14 15 19 19 17 14 11 12 20 19 18 14 13 15 18 13 0.707 IA-6 7 11 4 13 16 9 13 8 12 15 9 13 15 17 8 13 13 15 0.533 IA-7 16 21 16 9 18 14 18 21 16 20 13 11 13 21 7 16 15 19 0.717 IA-8 18 15 22 20 11 17 20 15 19 21 19 15 17 19 20 18 16 17 0.806 IA-9 4 7 18 5 0 1 5 11 4 5 5 9 3 8 1 6 1 6 0.250 IA-10 6 2 2 4 6 6 3 2 2 3 4 4 4 2 11 3 14 4 0.207 IA-11 9 14 9 8 8 10 15 17 15 13 11 8 14 16 15 9 6 16 0.538 IA-12 5 17 7 3 2 5 6 18 7 10 8 3 0 7 2 1 2 5 0.273 IA-13 3 3 5 7 9 3 4 4 3 4 3 2 10 3 3 2 12 3 0.210 IA-14 10 10 15 16 14 11 10 12 8 7 16 7 11 10 12 12 11 12 0.515 IA-15 12 8 17 19 17 16 19 19 17 11 15 18 20 12 18 20 10 20 0.727 IA-16 21 20 21 22 22 22 22 5 22 22 21 20 21 22 21 22 21 22 0.932 IA-17 1 4 11 2 12 4 1 3 5 6 1 10 5 5 0 7 4 2 0.210 IA-18 0 0 0 0 1 0 0 0 0 0 2 0 1 1 4 0 0 0 0.023 IA-19 19 5 19 18 15 20 11 10 13 9 18 22 19 13 16 17 19 9 0.687 IA-20 17 19 13 10 7 15 14 16 9 16 12 14 9 15 14 4 3 10 0.548 IA-21 14 9 3 12 10 18 8 20 10 8 6 6 6 6 6 10 17 8 0.447 IA-22 22 18 20 21 21 21 21 13 21 18 22 21 22 18 22 21 22 21 0.922 IA-23 15 22 8 6 13 12 12 22 20 17 17 6 7 9 10 19 9 18 0.611 30

2144-152- T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE APPENDIX E BRIEFING PROCEDURE They were not pushed to make a decision or judgment on these illustrative cases as the objective was merely to bring out explicitly the various conThe following is a copy of the instructions siderations that entered into the judgment. ) given each officer before he began ranking the stimuli: "There are some things you should know as background. "The problem with which we are concerned is the evaluation of surveillance systems and of com- "Regard yourself as a regimental commander in ponents of a surveillance system. an attack situation with normal capabilities. You can draw upon higher headquarters for support. "The end product of a surveillance system is an The artillery shown is direct support artillery. You intelligence estimate and this estimate is an ap- are free to choose whether to fight during the day or proximation to a true situation. We are going to the night. develop an equation by means of which we can measure how well one estimate approximates its true "However, even more important for our pursituation compared with how well another estimate poses is the attitude with which you approach these approximates its true situation. problems. Remember that you are evaluating the surveillance system or component that came upwith "However, there are many respects in which an that estimate of that true situation. Essentially you estimate can deviate from a true situation, e. g., can imagine that you are ranking the'systems' as to identification, strength. and location, and we need which one you would prefer to have on your team. to collect data from the judgments of experienced Furthermore, you should judge not only as the commilitary personnel which we can use to determine manding officer of the regimental combat team but the parameters in these equations. Thus the prob- you should also try to keep in mind the welfare of the lems which we will present to you for judgment have larger units of which you constitute a part. Thus, no "school" solution. You could regard what we're for example, a surveillance system may come up doing as trying to arrive at a school solution by com- with a gross over-estimate of a true situation, so bining the careful judgments made by the officers you decide to draw upon higher headquarters for best qualified, in experience and training, to make support. This means, of course, that this support them. is not available elsewhere where it may be needed more, although you yourself will be readily able to "The general procedure is as follows you will accomplish your mission. If such an estimate of a be shown a series of estimates paired off with their true situation were a good one we wouldn't need surrespective true situations, at first all on the same veillance systems, we would just imagine the enemy terrain. You will be asked to rank order from right had maximum capabilities and we would probably to left the pairs (an estimate and its true situation) lose a lot of campaigns by not fighting them. in order of how well the estimate approximates the true situation. "On the other hand, of course, a surveillance system should be penalized if it underestimates a "I will show you an example: (Two true situ- true situation because now you might run up against ations were shown, one each on terrain instances something for which you are not prepared and suffer I and II, and corresponding estimates were exhib- serious losses. ited. Both situations displayed infantry and artillery units. The officers were asked to consider "Now, no surveillance system is perfect. Each which estimate more closely approximated its re- one will almost certainly give you an inaccurate or spective true situations. One fitted closer to the incomplete picture of what's out there, and your infantry situation than the artillery situation and the problem is to decide how important the various kinds other the reverse. There was no obvious or clear of inaccuracies are, relatively by rank ordering, the choice and the officers were asked merely to discuss best approximations on down to the worst approximathe considerations involved in deciding between them. tions from right to left. 31

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE "We will first do this for infantry units alone "And finally the same thing will be done for on three different terrain instances and then com- estimates that have both infantry and artillery on bine them. them. "Then the same thing will be done for a sur- "Are there any questions?" veillance system which only reports artillery units. 32

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE APPENDIX F DEBRIEFING PROCEDURES Name Date QUESTIONNAIRE 1. Find out how much combat command experience. Record in terms of months, echelon, and theater. 2. Same for experience as S-2 or in a G-2 section. Record in terms of months, echelon, and theater. 3. Ask the officer how he would describe his basis for deciding which estimates were best. Information we would like is: a) Did he have a formula? If so, what is it? b) Did units (infantry and artillery) need to be pin-pointed to be counted as hits? Or did he just look at over-all strength? Or something in between? c) How did he weigh information on infantry as compared with artillery? 33

In

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE DISTRIBUTION LIST (Unclassified) Copy No. 1-4 Office of Assistant Secretary of Defense (R and D), Tech. Library Branch, Department of Defense, Washington 25, D. C. 5 Special Assistant, (R and D), Office of the Under Secretary of the Army, Department of the Army, Washington 25, D. C. 6 Director, Weapons System Evaluation Group, Department of Defense, Washington 25, D. C. 7 Office, Assistant Chief of Staff, Intelligence, Training Division, Attn: Lt. Col. Paul E. Doherty, Department of the Army, Washington 25, D. C. 10 Chief, Army Security Agency, GAS-24, RD and TE, Attn: ASA Liaison Officer, Department of the Army, Washington 25, D. C. 11 Chief, Army Security Agency, GAS-15, Attn: ASA Liaison Officer, Department of the Army, Washington 25, D. C. 12 Commanding General, U S. Army Combat Surveillance Agency, Department of the Army, Washington 25, D. C. 13 Office, Deputy Chief of Staff for Military Operations, Attn: Director, Plans Directorate, Department of the Army, Washington 25, D. C. 14 Office, Deputy Chief of Staff for Military Operations, 0 and T Directorate, Doctrines and Combat Developments Division, Department of the Army, Washington 25, D. C. 15 Office, Deputy Chief of Staff for Military Operations, O and T Directorate, Attn: Major Thomas R. Dolezal, Department of the Army, Washington 25, D. C. 16-17 Commanding Officer, Army Liaison Group (9550), Project MICHIGAN, Willow Run Laboratories, Ypsilanti, Michigan FOR TRANSMISSION TO Commander, British Joint Services Mission, 1800 K Street, N. W., Washington, D. C. THROUGH Office of the Chief Signal Officer, Attn: SIGRD-5-a, Department of the Army, Washington 25, D. C. 18 Chief, Tactics Division, Support Weapons Group, Operations Research Office, The Johns Hopkins University, 7100 Connecticut Ave., Chevy Chase 15, Maryland 19 Chief, Home Defense Division, Operations Research Office, The Johns Hopkins University, 7100 Connecticut Ave., Chevy Chase 15, Maryland 20 Lt. Col. A. W. Harris, USMC, Office, Assistant Chief of Bureau for Electronics (Code 803), Bureau of Ships, Department of the Navy, Washington 25, D. C. 21 Chief of Transportation, Attn: AVD-ED, Department of the Army, Washington 25, D. C. 22 Office, Chief of Research and Development, Attn: Lt. Col. W. M. VanHarlingen, Department of the Army, Washington 25, D. C. 23 Assistant Chief of Staff, G2, O and T Division, Attn: Lt. Col. R. V. Fridrich, U. S. Marine Corps Headquarters, Arlington Annex, Columbia Pike and Arlington Ridge Road, A rlington, Virginia 24 Chief, Aerial Reconnaissance Laboratory, Attn: Col. J. R. Knight, Wright Air Development Center, Wright-Patterson Air Force Base, Ohio 25-26 Commanding General, Redstone Arsenal, Huntsville, Alabama 35

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE 27 Commanding General, Frankford Arsenal, Attn: Chief, Research and Development Department, Philadelphia 37, Pennsylvania 28 Commanding General, Detroit Arsenal, Attn: Chief, Research and Development Division, Center Line, Michigan 29 Commanding General, Aberdeen Proving Ground, Maryland 30-35 Office, Chief of Ordnance, Research and Development Division, Department of the Army, Washington 25, D. C. 36 Commanding Generals Transportation Research and Development Station, Fort Eustis, Virginia 37-38 Chief of Engineers, Attn: ENGNF, Department of the Army, Washington 25, D. C. 39 Chief of Engineers, Attn: ENGIE, Department of the Army, Washington 25, D. C. 40 Chief of Engineers, Attn: ENGIS, Department of the Army, Washington 25, D. C. 41-42 Director, Engineer Research and Development Laboratories, Attn: Chief, Electrical Engineering Department, Fort Belvoir, Virginia 43 Director, Engineer Research and Development Laboratories, Attn: Chief, Topographic Engineering Department, Fort Belvoir, Virginia 44 Director, Engineer Research and Development Laboratories, Attn: Chief, Military Engineering Department, Fort Belvoir, Virginia 45 Commandant, The Engineer School, Fort Belvoir, Virginia 46 Office of the Quartermaster General, Building T-A, 2nd and T Streets, S. W., Department of the Army, Washington 25, D. C. 47-55 Chief, Research and Development Division, Office of the Chief Signal Officer, Department of the Army, Washington 25, D. C. 56 Commander, Chemical Corps Research and Development Command, Department of the Army, Washington 25, D. C. 57 W. J. Merchant, Chief Int. Div., Operations Research Office, The Johns Hopkins University, 7100 Connecticut Ave., Chevy Chase 15, Md. 58 Commanding General, Signal Corps Engineering Laboratory, Attn: Mr. H. P. Hutchinson, Fort Monmouth, New Jersey 59 Commanding Officer, Army Map Service, Attn: Documents Library, 6500 Brooks Lane, Washington 25, D. C. 60 Commanding General, Army Electronic Proving Ground, Attn: Lt. Col. S. H. Webster, Battle Area Surveillance Department, Fort Huachuca, Arizona 61 Commanding Officer, Human Research Unit Nr. 1, Attn: Dr. W. D. Boiers, Fort Knox, Kentucky 62 Commanding General, Army Ballistic Missile Agency, Attn: ORDAB-T, Huntsville, Alabama 63 Directors, Human Resources Res. Office, Attn: Security Officer, George Washington University, P. O. Box 3596, Washington 7, D. C. 64 Fred Thompson, c/o Combat Development Department, General Analysis Corporation, Fort Huachuca, Arizona 65 General Analysis Corporation, Attn: Alex Mood, Santa Monica, California 36

2144-152- T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE 72-96 Commanding General, Signal Corps Engineering Laboratory, Attn: SIGEL-DR, Fort Monmouth, New Jersey 97 President, Signal Corps Board, Fort Monmouth, New Jersey 98 Commander, Air Force Armament Center, Attn: Mr. Henry Maulshagen, Eglin Air Force Base, Florida 99 Haller,. Raymond and Brown, Attn: SCEAG, University Park, Pennsylvania 100 Commanding Officer, Signal Corps Electronics Research Unit, (9560), P. O. Box 205, Mountain View, California 101-102 Commanding General, Army Electronic Proving Ground, Attn: Chief, Battle Area Surveillance Department, Fort Huachuca, Arizona 103 Dr. W. R. G. Baker, Vice President and General Manager, Electronics Department, General Electric Company, Electronics Park, Syracuse, N. Y. 104 Aerojet-General Corporation, Attn: Librarian, P. 0. Box 296, Azusa, California VIA Bureau of Aeronautics Representative, c/o Aerojet-General Corporation, 6352 N. Irwindale Avenue, Azusa, California 106 Director, Research Studies Institute, Air University Command, Attn: Chief, ADTIC Maxwell Air Force Base, Alabama 107 Dr. Michael Ference, Jr., Ford Motor Company, P. O. Box 2053, Dearborn, Michigan 110 Dr. Albert G. Hill, Room 2E990, The Pentagon, Washington 25, D. C. 112 Dr. Andrew Longacre, Control Systems Laboratory, University of Illinois, Urbana, Illinois 117 Director, National Bureau of Standards, Attn: Division 12, Washington 25, D. C. 118-128 Commanding General, Continental Army Command, Fort Monroe, Virginia 129 Commanding General, Attn: CDTEC, Fort Ord, California 133 Commanding General, AA and GMC, Fort Bliss, Texas 134 Commanding General, The Armored Center, Fort Knox, Kentucky 135 President, Continental Army Command Board No. 6, Fort Rucker, Alabama 136 Commanding General, The Infantry Center, Fort Benning, Georgia 137 Assistant Commandant, The Artillery School, AA and GM Branch, Fort Bliss, Texas 138 Commandant, The Armored School, Fort Knox, Kentucky THROUGH Commanding General, The Armored Center, Fort Knox, Kentucky 139-140 Assistant Commandant, The Artillery and Guided Missile School, Fort Sill, Oklahoma 141 Commandant, The Infantry School, Fort Benning, Georgia 142-144 Commandant, Army War College, Carlisle Barracks, Pennsylvania 145-147 Commandant, Command and General Staff College, Fort Leavenworth, Kansas 148-149 President, Army Intelligence Board, Army Intelligence Center, Fort Holabird, Maryland 150 Commandant, Army Aviation School, Fort Rucker, Alabama 37

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE 151 Commandant, Joint Air Support School, c/o Commanding General, Continental Army Command, Fort Monroe, Virginia 152-153 President, Continental Army Command Board No. 1, Fort Sill, Oklahoma 154 President, Continental Army Command Board No. 2, Fort Knox, Kentucky 155 President, Continental Army Command Board No. 3, Fort Benning, Georgia 156 President, Continental Army Command Board No. 4, Fort Bliss, Texas 157 President, Continental Army Command Board No. 5, Fort Bragg, North Carolina 158 Army Section, Marine Corps Equipment Board, Quantico, Virginia 159 Commanding General, Continental Army Command, Attn: Col. William M. Slayden, GS (Armor), Fort Monroe, Virginia 160 Chief, Office of Naval Research, Building T-3, Constitution Avenue between 16th and 17th Streets, Department of the Navy, Washington 25, D. C. 161-162 Director, Naval Research Laboratories, 4th and Chesapeake Streets, S. W., Department of the Navy, Washington 25, D. C. 163 Chief of Naval Operations (OP-37), Department of the Navy, Washington 25, D. C. 164-168 Chief, Office of Naval Research, Building T-3, Attn: Col. W. N. Flournoy, USMC, Department of the Navy, Washington 25, D. C. 169-170 Commandant of the Marine Corps, Arlington Annex, Columbia Pike and Arlington Ridge Road, Arlington, Virginia 171-172 Director, Marine Corps Development Center, Quantico, Virginia 173 Commander, United States Air Force Security Service, Attn: OPL, San Antonio, Texas 174-175 Department of the Air Force, Headquarters USAF, Attn: AFDRQ, Washington 25, D. C. 176-177 Department of the Air Force, Headquarters USAF, Attn: AFDRD, Washington 25, D. C. 178-179 Department of the Air Force, Headquarters USAF, Attn: AFDAP, Washington 25, D. C. 180-181'Department of the Air Force, Headquarters USAF, Attn: AFOAC, Washington 25, D. C. 182-183 Department of the Air Force, Headquarters USAF, Attn: AFOOP, Washington 25, D. C. 184-185 Department of the Air Force, Headquarters USAF, Attn: AFOIN, Washington 25, D. C. 186-200 Commander, Wright Air Development Center, Attn: WCLRO (Staff), Wright-Patterson Air Force Base, Ohio 201-202 Commander, Air Force Cambridge Research Center, Cambridge, Massachusetts 203-204 Commander, Rome Air Development Center, Air Research and Development Command, Attn: RCSST-3, Rome, New York 205 Commander, Air Force Armament Center, Attn: ACOTT, Eglin Air Force Base, Florida 206 Director, Research Studies Institute, Air University Command, Attn: Chief ADTIC, Maxwell Air Force Base, Alabama 207-211 Documents Service Center, Armed Services Technical Information Center, Knott Building, Dayton 2, Ohio 212 Chief, Armed Forces Special Weapons Project, Attn: Adjutant General, Room 1B662, Pentagon Building, Washington 25, D. C. 213 Headquarters, USAF, ATTN: AFOIVI-3AIEI, Temop U Bldg., 12th and Constitution N. W., Washington 25, D. C. 38

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE 218-219 Commander, Headquarters Ninth Air Force, Attn: DCS/O, Shaw Air Force Base, South Carolina 220-221 Commander, Air Technical Intelligence Center, Wright-Patterson Air Force Base, Attn: Mr. R. B. Keeney, ATIR, Dayton, Ohio 222-223 Commander, Air University, Maxwell Air Force Base, Montgomery, Alabama 224-225 Commander, Air Proving Ground Command, Attn: Deputy for Operations, Eglin Air Force Base, Florida 228-229 Commander, Strategic Air Command, Offutt Air Force Base, Omaha, Nebraska 230-231 Commander, Far East Air Forces, APO 925, do Postmaster, San Francisco, California 232-233 CINC, United States Air Forces Europe, Attn: Director of Intelligence, APO 633, do Postmaster, New York, New York 234-235 The RAND Corporation, Attn: Mr. M. Davies, 1700 Main Street, Santa Monica, California 236-238 Headquarters, Tactical Air Command, Attn: Col. James L. Rose, Langley Air Force Base, Virginia 259-240 Commanding General, Air Research and Development Command, P.O. Box 1395, Attn: Lt. Col. J. J. Pellegrini, Baltimore, Maryland 241 Cornell Aeronautical Laboratory, Inc., Attn: Librarian, 4455 Genesee Street, Buffalo 21, New York THROUGH Bureau of Aeronautics Representative (Contract No. AF 18(600)-2), 4455 Genesee Street, Buffalo 21, New York 242 Pacific Division, Bendix Aviation Corporation, Attn: Mr. W. C. Leitch, Chief Electronics Engineer, 11600 Sherman Way, North Hollywood, California THROUGH Inspector of Naval Materiel, 1206 Santee Street, Los Angeles 15, California 243 Raytheon Manufacturing Company, Government Contracts Division, Attn: Documents Section, Waltham 54, Massachusetts THROUGH Assistant Inspector of Naval Materiel, Foundry Avenue, Waltham 54, Massachusetts 244 The Glenn L. Martin Company, Baltimore 3, Maryland THROUGH Air Force Plant Representative Office, WRAMA, The Glenn L. Martin Company, Baltimore 3, Maryland 245 Columbia University, Electronics Research Laboratories, Attn: Miss Helen K. Cressman, Technical Editor, 632 W. 125th Street, New York 27, New York THROUGH Commander, Rome Air Development Center, Attn: RCSST-3, Griffiss Air Force Base, Rome, New York 251 Commanding General, Detroit Arsenal, Attn: ORDMX-ECCT-Astron Project Engineer, Center Line, Michigan 39

2144-152-T THE UNIVERSITY OF MICHIGAN ENGINEERING RESEARCH INSTITUTE 252 Chief, Los Angeles Ordnance District, U. S. Army, Attn: ORDEV, Ord. Res. Br., 55 South Grand Avenue, Pasadena 2, California 253 General Electric Company, Attn: Miss Sophia Rugare, Librarian, French Road, Utica, New York THROUGH Office of the Air Force Plant Representative, MAAMA, General Electric Company, Electronics Park, Syracuse, New York 254 Commanding Officer, Army Liaison Group (9550), Project MICHIGAN, Willow Run Laboratories, Ypsilanti, Michigan 256 Office, Continental Army Command Liaison Officer, Project MICHIGAN, Willow Run Laboratories, Ypsilanti, Michigan 257 Corps of Engineers Liaison Officer, Project MICHIGAN, Willow Run Laboratories, Ypsilan Michigan 258 Air Force Development Field Representative, Project MICHIGAN, Willow Run Laboratories, Ypsilanti, Michigan 259 Commander, Wright Air Development Center, Attn: WCOL-9, Wright-Patterson Air Force Base, Dayton, Ohio 261-310 The University of Michigan, Internal Distribution 40

+I AD Accession No. UNCLASSIFIED AD Accession No. UNCLASSIFIED The University of Michigan, Engineering Research Institute, Willow Run Labora- The University of Michigan, Engineering Research Institute, Willow Run Laboratories, Ypsilanti, Michigan. tories, Ypsilanti, Michigan. Report of Project MICHIGAN, Evaluating Surveillance Systems, Caldwell, W. V., Report of Project MICHIGAN, Evaluating Surveillance Systems, Caldwell, W. V., Coombs, C. H., and Thrall, R. M. 1. Surveillance System Coombs, C. H., and Thrall, R. M. 1. Surveillance System Model Model Report No. 2144-112-T, Jane 1957, 36 pp., 4 illus., Project 2144 (Contract Report No. 2144-152-T, June 1957, 36 pp., 4 illus., Project 2144 (Contract 2 Testing of Model Report No. 2144-152-T, June 1957, 36 pp., 4 illus., Project 2144 (Contract ontract2 DA-36-039 SC-52654, DA Project NR-3-99-10-024, Sig C No. 102D), UNCLASSIFIED e DA-36-039 SC-52654, DA Project NR-3-99-10-024, Sig C No. 102D), UNCLASSIFIED. esting of Model 3. Evaluation of Tests 3. Evaluation of Tests The first phases of a long range project for developing models for evaluating intelli- 4. Contract DA-36-039 The first phases of a long range project for developing models for evaluating intelli- 4. Contract DA-i-9 gence systems in terms of measurable outputs can now be reported. In making such gence systems in terms of measurable outputs can now be reported. In making such evaluations, the number of correct identifications, number of false alarms, number SC5i4gnestmsstrsofeaublotpscnnwbeeotd.Imasguh evaluations, the number of correct identifications number of false alarms, number evaluations, the number of correct identifications, number of false alarms, number SCof displacements, and number of complete misses must be measured, in each of of displacements, and number of complete misses must be measured, in each of several categories (such as infantry, artillery, armor, supply lines), each of a several categories (such as infantry, artillery, armor, supply lines), each of a number of sectors (front line, rear, far rear) and in each of several situations number of sectors (front line, rear, far rear) and in each of several situations (such as attack or defense, good or poor mobility). The major results to date are 1) (such as attack or defense, good or poor mobility). The major results to date are 1) the development of experimental techniques, 2) the formulation of an analytical the development of experimental techniques, 2) the formulation of an analytical model, 3) the development of analysis techniques for evaluation of parameters in the model, 3) the development of analysis techniques for evaluation of parameters in the model in terms of empirical results, 4) the preliminary evaluation of parameters model in terms of empirical results, 4) the preliminary evaluation of parameters (rank order correlations of 0. 8 and higher have been established between some of (rank order correlations of 0. 8 and higher have been established between some of the analytic predictions and the empirical results). and 5) formulation of several the analytic predictions and the empirical results). and 5) formulation of several concrete unsolved problems out of the over-all problem. Plans for continued study concrete unsolved problems out of the over-all problem. Plans for continued study are discussed in Sections 8.2 and 8.3. are discussed in Sections 8.2 and 8.3. UNCLASSIFIED UNCLASSIFIED! -~~~4. AD Accession No. UNCLASSIFIED AD Accession No. UNCLASSIFIED The University of Michigan, Engineering Research Institute, Willow Run Labora- The University of Michigan, Engineering Research Institute, Willow Run Laboratories, Ypsilanti, Michigan. tories, Ypsilanti, Michigan. Report of Project MICHIGAN, Evaluating Surveillance Systems, Caldwell, W. V., Report of Project MICHIGAN, Evaluating Surveillance Systems, Caldwell, W. V., Coombs, C. H., and Thrall, R. M. 1. Surveillance System Coombs, C. H., and Thrall, R. M. 1. Surveillance System Model Model Report No. 2144-152-T, June 1957, 36 pp., 4 illus., Project 2144 (Contract 2 T Report No. 2144-152-T, June 1957, 36 pp., 4 illus., Project 2144 (Contract DA-36-039 SC-52654, DA Project NR-3-99-10-024, Sig C No. 102D), UNCLASSIFIED DA-3.-39 SC-2i4, DA Project NR-3-99 —24, Sig C No. 102D), UNCLASSIFIED. esting of Model 3. Evaluation of Tests 3. Evaluation of Tests ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~The first phases of a long range project for developing models for evaluating intelliThgene syrst phases of long rmeasurablnge project for developing models forted. In maluating intelli- 4 Contract DA-36-039 The first phases of a long range project for developing models for evaluating intelli- 4 Cntt DA-3-039 evalua tions, the numb terms of c orrec t identifipucations nowumber of false Inalarm s, n umbrch SC-24 gence systems in terms of measurable outputs can now be reported. In making such, -5 evaluations, the number of correct identifications number of false alarms, number evaluations, the number of correct identifications. number of false alarms, number of displacements, and number of complete misses must be measured, in each of of displacements, and number of complete misses must be measured, in each of several categories (such as infantry, artillery, armor, supply lines), each of a several categories (such as infantry, artillery, armor, supply lines), each of a number of sectors (front line, rear, far rear) and in each of several situations number of sectors (front line, rear, far rear) and in each of several situations (such as attack or defense, good or poor mobility). The major results to date are 1) (such as attack or defense, good or poor mobility). The major results to date are 1) the development of experimental techniques, 2) the formulation of an analytical the development of experimental techniques, 2) the formulation of an analytical model, 3) the development of analysis techniques for evaluation of parameters in the model, 3) the development of analysis techniques for evaluation of parameters in the model in terms of empirical results, 4) the preliminary evaluation of parameters model in terms of empirical results, 4) the preliminary evaluation of parameters (rank order correlations of 0.8 and higher have been established between some of (rank order correlations of 0.8 and higher have been established between some of the analytic predictions and the empirical results). and 5) formulation of several the analytic predictions and the empirical results). and 5) formulation of several concrete unsolved problems out of the over-all problem. Plans for continued study concrete unsolved problems out of the over-all problem. Plans for continued study are discussed in Sections 8.2 and 8.3. are discussed in Sections 8.2 and 8.3. UNCLASSIFIED UNCLASSIFIED 4

AD Accession No. UNCLASSIFIED AD Accession No. UNCLASSIFIED The University of Michigan, Engineering Research Institute, Willow Run Labors- The University of Michigan, Engineering Research Institute, Willow Ron Laboratories, Ypsilanti, Michigan. tories, Ypsilanti, Michigan. Report of Project MICHIGAN, Evalusting Sorveillance Systems, Caidwell, W. V., Report of Project MICHIGAN, Evaiuating Surveillance Systems, Caldwell, W. V., Coombs, C. H., and Thrall, N. M. 1. Sorveillance System Coombs, C. H., and Thrall, N. M. 1. Sorveillance System Model Model Report No. 2144-112-T, Jose 1917, 39 pp., 4 titus., Project 2144 (Contract DA-3t-039 SC-152J4, DA Project NR-3-99-6t-024, Sig C No. lt2D), UNCLASSIFIED 2. Testing of Model Report No. 2144-152-T, June 1957, 39 pp., 4 illus., Project 2144 (Contract DA-36-039 SC-52654, DA Project NR-3-99-l0-024, Sig C No. 102D), UNCLASSIFIED 2, Testing of Model 3. EvaluatIon of Tests 3. Evaloation of Tests The first phases of a long range project for developing models for evaloating intelli- The first phases of a long range project for developing models for evaluating intelligence systems in terms of measurable outputs can now be reported. In making such SContrac DA-3e-t3p d4. Contract DA-9 -m39 evaloations, the number of correct identifications, number of false alarmo, number SC-52954 gence systems so terms of measurable outputs can now be reported. In malsing such S-2i evaluations, the d number of correct identifications, number of false alarmsnumber evaluations, the number of correct identifications, number of false alarms, number of displacements, and number of complete misses most be measured, in each of of displacements, and number of complete misses most be measured, in each of several categories (such as infantry, artillery, armor, sopply lines), each of a several categories (such as infantry, artillery, armor, supply lines), each of a number of sectors (front line, rear, far rear) and in each of several situations number of sectors (front line, rear, far rear) and in each of several situations (such as attach or defense, good or poor mobility). The major results to date are 1) (such an attach or defense, good or poor mobility). The major results to date are 1) the development of experimental techniques, 2) the formulation of an analytical the development of experimental techniques, 2) the formulation of an analytical model, 3) the development of analysis techniques for evaluation of parameters in the model, 3) the development of analysis techniques for evaluation of parameters in the model in terms of empirical results, 4) the preliminary evaluation of parameters model in terms of empirical results, 4) the preliminary evaluation of parameters (rank order correlations of 9. 9 and higher have been established between some of (rank order correlations of 9. 9 and higher have been established between some of the analytic predictions and the empirical results), and 5) formulation of several the analytic predictions and the empirical results). and 5) formulatIon of several concrete unsolved problems out of the over-all problem. Plans for continued study concrete unsolved problems out of the over-all problem. Plans for continued study are discussed in Sections 8. 2 and 8. 3. are discussed in Sections 8. 2 and 8. 3. UNCLASSIFIED UNCLASSIFIED AD Accession No. UNCLASSIFIED AD Accession No. UNCLASSIFIED The University of Michigan, Engineering Research Institute, Willow Run Labors- The University of Michigan, Engineering Research Institute, Willow Run Laboratories, Ypsilanti, Michigan. tories, Ypsilanti, Michigan. Report of Project MICHIGAN, Evaluating Surveillance Systems, Caldwell, W. V., Report of Project MICHIGAN, Evaluating Surveillance Systems, Caldwell, W. V., Coombs, C. H., and Thrall, R. M. 1. SurveIllance System Coombs, C. H., and Thrall, R. M. 1. Surveillance System Model Md Report No. 3144-152-T, Jose 1957, 39 pp., 4 illus,,. Project 3144 (Contract oe DA-36-03R SC-152J4, DA Project NR-3-99-2t-144, Sig C No. lt3D), UNCLASSIFIED 2. Testing of Model Report No. 2144-152-T, June 1957, 39 pp., 4 illus., Project 2144 (Contract DA-36-039 SC-52654, DA Project NR-3-99-10-024, Sig C No. 102D), UNCLASSIFIED 3. T 3. Evaluation of Tests 3. Evaluation of Tests The first phases of a long range project for developing models for evaluating intelli- 4. Contract DA-36-039 The first phases of a long range project for developing models for evaluating intelli- 4 C genc sytemsin erm of easrabl ouput cannowbe rpored.In mbin suc *ConractDA-9-03 Th fist pase of log rnge rojct fr dvelpingmodls fr ealutingintlli evaluations, the number of correct identifications, number of false alarms, number SC-52654 gence systems in terms of measurable outputs can now be reported. In making such SC-52i 54 evaluations, thed number of correct identifcationsnumber of false alarms, number evaluations, the number of correct identifications, number of false alarms, number of displacements, and number of complete misses must be measured, in each of of displacements, and number of complete misses must be measured, In each of several categories (such as infantry, artillery, armor, supply lines), each of a several categories (such as infantry, artiliery. armor, supply lines), each of a number of sectors (front line, rear, far rear) and in each of several situations number of sectors (front line. rear, far rear) and in each of several situations (such as attach or defernse, good or poor mobility). The major results to date are 1) (ouch as attach or defense, good or poor mobility). The major results to date are 1) the development of experimental techniques, 2) the formulation of an analytical the development of experimental techniques, 2) the formulation of an analytical model, 3) the development of analysis techniques for evaluation of parameters in the model, 3) the development of analysis techniques for evaluation of parameters in the model in terms of empirical results. 4) the preliminary evaluation of parameters model in terms of empirical results, 4) the preliminary evaluation of parameters (rank order correlations of 9. 9 and higher have been established between some of (rank order correlations of 5. 9 and higher have been established between some of the analytic predictions and the empirical results). and 5) formulation of several the analytic predictions and the empirical results). and I) formulation of several concrete unsolved problems out of the over-all problem. Plans for continued study concrete unsolved problems out of the over-all problem. Plans for continued study are discussed in Sections 8. 2 and 8. 3. are discussed in Sections 8. 2 and 8. 3. UNCLASSIFIED UNCLASSIFIED + ~~~~~~~~~~~~~~-4

uI'01Vnl T 989 IvIInlUl5li 3 9015 02960 9891