Bureau of Business Research Graduate School of Business Administration University of Michigan April, 1971 I MULTIDIMENSIONAL AND UNIDIMENSIONAL METRIC SCALING OF PREFERENCE FOR JOB DESCRIPTIONS Working Paper No. 37 by Raymond E. Hill Assistant Professor of Organizational Behavior and Industrial Relations University of Michigan Edgar A. Pessemier Professor of Industrial Administration Krannert Graduate School of Industrial Administration Purdue University FOR DISCUSSION PURPOSES ONLY None of this material is to be quoted or reproduced without the express permission of the Bureau of Business Research

ABSTRACT The primary thrust of this paper is an exploration of two models for representing preference for job descriptions and the possible applications of the models in a computer-assisted personnel search system. The models are based on the premise that preference for a particular employment description is related to the distance between that offer and an ideal offer in a decision space. One model is a unidimensional decision space based on comparative preference judgments, whereas the other model is a multidimensional decision space constructed with the use of comparative similarity judgments between the job descriptions. The predictive power of each model is examined and related to two types of error. The first types of error are goodness-of-fit measures between the models and the data. The second types of error are unreliabilities associated with obtaining the data. Although some error measures were significantly related to the models statistically, it did not appear that either error type adversely affected the models in a practical sense. The possible application to computer-assisted personnel search involves the attempt to locate a given firm's employment offer(s) in the decision space of a given decision maker and to make inferences prior to search regarding the acceptability of the offer to the decision maker. BACKGROUND OF THIS PAPER This research was conducted at Purdue University's Krannert Graduate School of Industrial Administration and is part of Professor Hill's doctoral dissertation. Professor Pessemier served on the dissertation committee, and made substantial contributions to the substance of the dissertation as well as to this article. Future research regarding job choice processes will be conducted at the University of Michigan's Graduate School of Business Administration. This paper is being submitted for publication in the Industrial and Labor Relations Review.

This study compares two methods for preference scaling and examines the predictive power of each method in relation to selected measures of error. The objects which were scaled were a set of nine uniformly described employment offers and a tenth object referred to as the ideal offer. The procedures for preference scaling can be viewed as models which assign values to the job offers. Finally, these values are employed to predict job choice. Some implications for computer assisted man-job matching systems will be pointed out in the conclusions. An important aspect of the present study is the use of a subjectively defined "ideal" job offer. It is scaled along with nine other uniformly described offers. The motivation for this work stems originally from a 1967 study by Peer 0. Soelberg in which he examined the process of choosing jobs among graduates of MIT's Alfred P. Sloan School of Management, Among the conclusions of the study Soelberg notes: "The decision maker defines his career problem by deriving an ideal solution to it, which in turn guides his planning on a set of operational criteria for evaluating specific job alternatives. ",1/ Therefore, the 1/ Peer 0. Soelberg, "UJnprogrammed Decision Making, " Industrial Management Review, VIII (Spring 1967), 22. -1 -

-2 - notion of an ideal job offer is relevant to the choice process under inve s tigation. Recent developments in psychometric methods have resulted in some rather powerful tools for locating a point or region most preferred in a decision (joint) space. The ideal job offer will be used to predict preference and to study the relation of the model's predictive power to selected measures of error. The standardized employment offers are shown in Appendix A. They were presented to ninety 1968 candidates for the Master of Science Degree in Industrial Administration (MSIA) in the Herman C. Krannert Graduate School of Industrial Administration at Purdue University. The standard job offers were constructed on the basis of research conducted by Joseph Ullman on the previous three MSIA classes. The data in Ullman's research included approximately 1, 000 actual descriptions of employment offers and were reviewed in an attempt to write standard descriptions which would reflect real, salient characteristics. In addition, six dimensions of job choice were used in constructing the standard offers: 1. Opportunity for advancement 2. Challenge in the job 3. Content of the job 4. Salary 5. Geographic location 6. Prestige of the position These dimensions were consistently rated as important in the process of job choice by the subjects of Ullman's research. To increase the perceivable contrasts among the objects of choice, the job offers varied substantially along the above dimensions.

-3 - The concern of this study is the relative efficacy of two preference scaling procedures as applied to the set of nine standard job descriptions. Multidimensional and unidimensional scaling procedures were used to scale preference. These two methods were explored because they involve comparative judgments between alternative offers and an ideal —a process which Soelberg indicated is central to job choice. Multidimensional Scaling as a Procedure for Quantification of Preference The multidimensional scaling of preference for an individual subject entails the development of a metric configuration of t points in an n-dimensional space where t-l of the points are said to represent objects or stimuli (i. e., job descriptions in the present case) and the th t point represents the individual. The configuration which represents both objects and an individual jointly in the same space is referred to 2/ as a joint space.- The point in the joint space which represents the individual is referred to as the ideal point and is assumed to be the location of the ideal or most preferred object of all possible object locations in the space. In other words, the individual is said to occupy the position in the space which represents that object which he would prefer to all other possible objects. The subject's preference for each object in the space is an inverse function of the distance 2/ C. H. Coombs, A Theory of Data (New York: John Wiley & Sons, Inc., 1964).

-4 -from his ideal point to each object's point in the same space. The smaller the distance between an object and the ideal point, the more it is preferred. A two dimensional representation of a joint space is shown in Figure 1. The objective of the multidimensional scaling routine is to determine the coordinates of all the objects as well as the ideal point on the axes of the space. Objects A, B, and C might represent different job offers and the asterisk would represent the ideal job or point in that space. Object A is closest to the ideal point and would thus be most preferred, while object C is least preferred because it is farthest from the ideal. Distance in the joint space between the ideal point and any object, say object A, can be computed by the general Minkowski r distance formula (1). - /r In (1) dIA =.1 ( Ii- ADl)r Li 1 where d =distance between object A and ideal point IA A. = coordinate of object A on axis i I. = coordinate of ideal point on axis i r = the Minkowski number When distance in a joint space is computed with a Minkowski r equal to one, a "city block" metric is employed. Distances are then simply the sum across all axes of the positive differences between the coordinate values of the ideal point and the object on the axes,

-5 - * 1. I q A B0 Ic Fig. 1. Two-dimensional joint space.

-6 - A Minkowski r equal to two yields Euclidean distance since formula (1) reduces to an "n" dimensional extension of the Pythagorean Theorem from plane geometry. Figure 1 illustrates the difference between the city block and Euclidean spaces. The solid arrow between I and A represents Euclidean distance, whereas the dotted arrows represent city block distance. Minkowski numbers larger than two are difficult to interpret psychologically, but lead to increasing dominance of the distance by axes along which larger differences appear. It might be noted in passing that the axes or dimensions of the space in Figure 1 are not labeled or interpreted. It may or may not be useful to attempt to identify the axes of a joint with specific physical or psychological characteristics of the objects. If the researcher's concern is the quantification and prediction of preference, then it is not necessary to interpret the axes. If he is concerned with preference, and if verbal labeling of independent characteristics is feasible, he should take steps to infer the names of the axes. Additional analyses may be required to perform the latter task. Procedure for Developing the Joint Space There are several multidimensional scaling algorithms which can be used to develop joint space configurations. The method used 3/ in the present study is based on the Kruskal algorithm.- The input 3/ J. B. Kruskal, "Multidimensional Scaling by Optimizing Goodness of Fit to a Nonmetric Hypothesis, Psychometrika, XXIX (March 1964), and J. B. Kruskal, "Nonmetric Multidimensional Scaling: A Numerical Method, " Pychometrika, XXIX (June 1964).

-7 -data are similarity judgments for all possible pairs of job descriptions (t-l objects) including the subject's personal concept of his ideal job (t object). Appendix B shows examples of some of the job descriptions paired with each other as well as with the subject's ideal job. Associated with each pair is a scale on which the subject rates the similarity of the two jobs. The number 1 represents "very similar" and the number 13 represents "very dissimilar. " Since there are ten jobs (nine standards and one ideal), forty-five similarity ratings (t(t-l) /2 where t = 10) are the data from which each joint space configuration could be derived. Note that the subject never explicitly states the characteristics of his ideal job. He simply holds the ideal in his mind and makes comparative similarity judgments between it and all other job descriptions. The judged similarity between any two job offers is an index of the psychological distance between them. Two jobs that are judged very similar are psychologically close together, whereas two jobs judged very dissimilar are psychologically far apart. Therefore, psychological distance is one of the basic constructs upon which 4/ multidimensional scaling is based.- It should be emphasized, however, that the distance between real objects does not imply differences in degree of preference. Although objects that are very similar will always tend to be similarly preferred, some objects 4/ R. N. Shepard, "The Analysis of Proximities: Multidimensional Scaling with an Unknown Distance Function I, " Psychometrika, XXVII (June 1962).

-8 - that are very dissimilar may also be similarly preferred. For example, one may like both hot and cold consumme. A short illustrative example should serve to explicate what the Kruskal multidimensional scaling method and program does with the comparative similarity judgments. Suppose the similarity ratings shown in Table I were made on the six pairs of four objects, A, B, C and D, and that the rating scale was the same as that shown in Appendix B. The objects could be job offers or any other stimuli which were to be multidimensionally scaled. The following question can be posed: What is the minimum number of dimensions necessary to represent these "psychological distances? "-/ The answer is that this pattern of similarity judgments can be represented in a minimum space of two dimensions because there is no difference in the order of magnitude of the distance in the configuration in Figure 2 and the order of the similarity ratings or the psychological distances in Table 1. Thus, the nonmetric multidimensional scaling routine starts with a vector of nonmetric ordinal similarity measures and transforms them into a vector of metric interobject distances, d..s, in a space of some specified number of dimensions (two in this illustration). In the process of scaling the objects, order relationships are all that is required, but goodness of fit can be assessed 5/ H. Gullicksen, "Linear and Multidimensional Scaling," Psychometrika, XXVI (March 1961), 15.

-9 - Table 1. Example of Paired Similarity Judgments on Four Objects. Order of Order of Dissimilarity. onfigural Pair Dis similarity C onfigural Rating Distance s RRatingg. Distances Ditn AB 5 1 1 6.0 AC 12 6 6 13.8 AD 4 10 4 4 0 BC 6 2 2 7.8 BD 9 3 3 8.0 CD 11 5 5 11.2 Similarity ratings are referred to as dissimilarities when larger numbers on the scale represent increasing dissimilarity.

-10 - 00001 0 asc lP,r% I\ I Fig. 2, Multidimensional scaling solution with associated configural distances.

-11 -by comparing the scaler of the configuration to the initial data. If a fifth object representing a person's ideal object had been scaled, then Figure 2 would be a two-dimensional joint space. The present Figure 2 represents only a perceptual map of how the subject "sees" the real objects. The output of the Kruskal program is a matrix of coordinates in which the rows represent objects and the columns represent axes of the space. The selection of the origin in the space is arbitrary. The Kruskal routine sets the origin near the middle of the configuration of points by constraining the root mean square of the loadings on the 6/ axes to be one. — Since there is a good fit between the pattern of similarity judgments in Table 1 and the interobject distances of Figure 2, the inference would be that the subject was primarily using two dimensions in his perceptual judgments about the four objects. Kruskal suggests a measure of goodness of fit which reflects the fit of the original similarity judgments, S..s, and the derived interobject distances, d..s (i <j; i, j = I,...., n; n objects). This measure is called 1j stress. It is calculated from the least squares fitted values, d.., that 1J are monotone with the order of interobject similarity measures, S... 1j 6/ Kruskal, "Multidimensional Scaling. " ii i ii i i i I I i I i i I I Ii i i i I i i i I 1 i I I I I i

-12 - i< j ij ij (2) S - --- --- _ — 2 d icj ij where d.. = the value on the monotone least squares curve which minimizes S. 7/ The derived configurations or multidimensional scaling solutions and related levels of stress can be obtained for any number of dimensions less than the number of objects. As the number of dimensions increase, the fit of the configuration distances to the observed distances improves and the level of stress declines. Stress is thus defined in a "configura8/ tion space. "- Kruskal gives the following "verbal evaluations" to selected levels of stress: Stress Goodness of Fit.20 Poor.10 Fair 05 Good 025 Excellent 00 Perfect Predictability of Joint Spaces Joint spaces were developed for ninety subjects by having each 7/ The April 1964 version 1 Kruskal program was used in this study. In March 1968 expanded versions (version 4 and 4M) became available in which T was subtracted from d. in the denominator before ij squaring. 8/ Kruskal, "Nonmetric Multidimensional Scaling."

-13 -subject complete all forty-five paired similarity judgments on the ten job descriptions. Three days after completing these similarity judgments, the subjects were asked to rank the job descriptions in order from most to least preferred. This actual rank order was then used to check- the accuracy with which the joint spaces captured the preference ordering of the job descriptions for each subject. That is, a derived rank order was obtained from each joint space on the basis of the distance of each description from the ideal, and the derived rank order was correlated with the actual rank order by using the Spearman rho or rank order correlation coefficient. Rho measures the predictive power of distance in the joint space between a job and the subject's ideal job. 6Cd (3) rho = 1 - n3 -n where d = difference between derived and actual ranks for any given job n = number of observations (nine in this case) An example of a two-dimensional Euclidean joint space for Subject X is shown in Figure 3. Table 2 shows the distance from the ideal for nine job descriptions and the derived and actual rank 2 orders and intermediate calculations of d and d for the computation of rho. In this case rho is. 98, which is highly significant since a rho of. 78 is significant at the 1 per cent level and a rho of. 60 is 9/ significant at the 5 per cent level.- The stress value associated 9/ S. Siegel, Nonparametric Statistics for the Behavioral Sciences (New York: McGraw-Hill, Inc., 1956), p. 284.

-aoseds 4uioC upieapprina ieuoTsuauiTp-oA& jo aldwuexa *E 2 eSi O'1 - a~b Qfl A R Rl g0l., I -..l -q I1' I.. ' III I IS lllmll -~~~~~~- -ii *A Adlkl 4 ) ~I 8 01 %A

0 Table 2. Distances of Nine Jobs from the Ideal Point and Derived and Actual Ranks in Two-Dimensional Euclidean Space for Subject X. Distance from Derived Actual 2 Ideal Point Rank Order Rank Order 1 1.46 6 6 0 0 2 2.22 7 8 1 3 1.08 4 4 0 0 4 2.27 8 7 1 5 1.31 5 5 0 0 6.42 2 2 0 0 7 2. 34 9 9 0 0 8.14 1 1 0 0 9.83 3 3 0 0 Ed2 2 rho =.98 tl I

-16 -with the two-dimensional Euclidean joint space for Subject X was. 079. Examination of the predictive power of the preference distances in the joint spaces as well as the stress patterns indicated that a four-dimensional Euclidean joint space was appropriate for the analysis at hand when applied to all ninety subjects in the study. Four dimensions provided a reasonably parsimonious representation of the original data but sacrificed little in terms of the average predictability across subjects (average Spearman rho). Table 3 shows selected indices of predictability for the four-dimensional Euclidean joint spaces across ninety subjects. The prediction of preference on the basis of multidimensionally scaled similarity data is very encouraging. In addition, these results empirically support the theory that job preference involves comparative judgments 10/ against an ideal as Soelberg has hypothesized. The Dollar Metric Unidimensional Scale as a Predictor of Preference The Dollar Metric unidimensional scaling procedure was developed by Edgar A. Pessemier. It places objects or jobs on one dimension —dollar unit scale. The following example is patterned 10/ Soelberg, "Unprogrammed Decision Making." 11/ E. Pessemier, "Measuring Social, Scientific and Military Benefits in a Dollar Metric," Institute Paper No. 146, Aug., 1966; E. Pessemier and R. Teach, "A Single Subject Scaling Model Using Judged Distance between Pairs of Stimuli," Institute Paper No. 143, Aug., 1966; and E. Pessemier and R. Teach, "Unidimensional Scaling of Individual and Group Preference, " Institute Paper No. 282, July, 1970. All three papers issued by the Institute for Research in the Behavioral, Economic and Management Sciences, Herman C. Krannert Graduate School of Industrial Administration, Purdue University, Lafayette, Indiana.

-17 - i I i i i i II Ii i i i I i i i I i i i i iI i I I i Table 3. Spearman Rhos Based on Four-Dimensional Euclidean Joint Spaces for Ninety Subjects. Subject Rho Subject Rho Subject Rho 1.; 85 31.47 61.92 2.68 32.87 62.77 3.78 33.88 63.83 4.82 34.77 64.95 5.70 35.82 65.35 6.42 36.75 66.63 7.97 37.95 67.58 8.83 38.90 68.88 9.98 39.35 69.67 10.92 40.92 70.00 11.78 41.73 71.82 12.87 42.88 72.83 13.42 43.75 73.28 14.82 44.48 74.43 15.87 45.85 75.20 16.87 46.83 76. 88 17.82 47.35 77.63 18.68 48.25 78.75 19. 05 49.52 79.95 20.33 50.67 80.53 21.38 51.58 81.88 22 -.32 52.85 82.68 23.90 53.83 83.73 24.88 54.87 84.22 25.78 55.75 85.95 26.98 56.93 86.77 27.72 57.63 87.52 28.88 58.42 88.83 29.85 59.43 89.88 30.38 60.83 90.78 Summary 1 per cent significance (No,. 78) 46 (51 per cent) 5 per cent significance (No.>. 60) 65 (73 per cent) Number of first choices predicted 43 (48 per cent) Mean rank correlation.69

"' Ill.. -18 -after the explanation by Pessemier and Teach. The basic data are paired comparative judgments like those shown in Appendix C. The first page of Appendix C shows the instructions for making the judgments. A subject selects from each pair the job he prefers and the number of dollars per month which would have to be added to the other job to make him change his original preference. This judgment is an index of the strength of the subject's preference for one job 12/ over the other. In the notation of Pessemier and Teach,- d'ij ijk represents this subjectively evaluated preference of job i over job j (in dollars) for the k subject. Since there are ten jobs (nine plus the one ideal), the subject is required to make forty-five comparative preference judgments. Two assumptions are made in using these original dollar judgments for calculating scale values for the jobs. The first is that preference is symmetrical, so dP = -d' all i and j, i j ijk jik' The symmetry assumption means that if subject k was asked to indicate how many dollars per month would have to be subtracted from the salary of the preferred job to make him change his original preference, his response would be the same for the job pair i and j. Since the reversed experimental procedure was not carried out, the validity of the symmetry assumption has not been tested. 12/ Pessemier and Teach, "Unidimensional Scaling of Individual and Group Preference, " ibid.

-19 - The second assumption is that the preference for a job compared to itself is zero. This can be shown as follows: d', = 0 all i = j ijk The scale values for each job are calculated by arranging the d'ijk for a particular subject into a square matrix with ten rows and ijk ten columns. The columns of the matrix are averaged. The average of the di s in the first column of the matrix is the scale value for the ijk first job. The average of the second column is the scale value for the second job and -so forth for all ten columns. (The average of the tenth column is a scale value for the ideal job. ) This procedure was carried out ninety times to derive the Dollar Metric scale values for all ten jobs for each of the ninety subjects. The matrix of original judgments (d'iks) for Subject X and the ijk | ^. Dollar Metric scale values of the jobs are shown in Table 4. In this particular preference matrix the columns are arbitrarily defined to represent the preferred jobs and the rows to designate the nonpreferred jobs. The preference matrix is skew symmetric since d'ij = -d'j ijk jik The data (d'.. s) are coded into the preference matrix by letting i ijk represent the column and j the row. As an illustration, Subject X judged Job 3 to be preferred to Job 1 by $100. Therefore, d' 31X equals 100 and a positive 100 would then be entered in column 3, row 1 or the matrix. In column I, row 3, a negative 100 would be entered according to the symmetry assumption. The same rationale applies to all forty-five d'i s and the related -d'j. s. Also note that the ijk jik

t i t Table 4. Preference Matrix and Dollar Metric Scale Values of Ten Jobs for Subject X. Job 1 Job 2 Job 3 Job 4 Job 5 Job 6 Job 7 Job 8 Job 9 Job 10 Job 1 Job 2 0150 Job 3 -0100 Job 4 0300 Job 5 0100 Job 6 -0200 Job 7 0300 Job 8 -0300 Job 9 -0300 Job 10 -0300 -0150 0100 0300 -0300 0095 0200 -0450 0200 -0400 -0200 0100 0400 -0400 -0300 -0200 -0200 -0600 -0250 -0300 -0095 -0200 -0400 -0350 -0200 -0485 0300 -0400 -0100 0450 -0200 0400 -0200 0400 -0200 0025 -0750 0200 0400 0200 0350 0200 0500 -0100 -0100 -0150 -0300 0300 -0100 0400 -0400 0300 -0200 0485 -0400 0200 -0500 0100 0500 -0500 0397 0150 -0565 -0075 0300 0200 0200 0300 -0025 -0100 0397 -0150 0300 0600 0250. 0400 0750 0150 0565 0075 0200 -0200 I O bum of columns _ 350 -2305 250 -2130 - 175 1500 -2568 2360 922 3290 Scale values, 8!= I average of column - 35 0 where: n = 10 - 230. 5 25.0 - 213.0 - 17. 5 150.0 - 256.8 236.0 92.2 329. 0

-21 -diagonal elements of the matrix are all zero in accordance with the assumption that d' = 0. iik The average of each column of the matrix is a least squares estimate of the relative preference of the given job compared to all remaining jobs. Furthermore, the average of a given column, column A i, is the Dollar Metric scale value for job i and is referred to as 0C.' ik where k represents the subject. Thus jobs with predominantly positive values in the column will have high positive scale values (i.e., Job 8, or column 8 of Table 4), whereas jobs with predominantly negative values (i. e., Job 7) will have low negative scale values. Jobs with a mix of positive and negative numbers associated with them will have scale A values closer to zero, the scale centroid. It is also true that the o'- s ik are least squares estimates of the "true" scale values, oC l s, since ik /k the ol 's are chosen to minimize ik i A 2 (4) j (di where d' = raw subjective preference of job i over job j ijk A A j' c Lj-k' = bderived preference scale separation of job i and job j Although the preference scale distance, 'ij', between jobs i and j corresponds as closely as possible, in the sense of least squares, to the original data d.., these are interval measures without a natural 1j origin. To provide a natural origin, a scale anchor is required. It could be the lowest salary at which the subject would accept the least

-22 -preferred job (when all the dollar metric scale differentials in salary prevailed). These anchors were not estimated in this study. Table 5 shows the rank order derived from the Dollar Metric scale and the actual rank order for Subject X. The Spearman rho between the derived and actual ranks is 1. 00, which is perfect rank correlation for Subject X. The Spearman correlations between derived and actual ranks are shown in Table 6 along with the summary statistics for all but three subjects. Three subjects provided incomplete data so scale values could be calculated for only eighty-seven subjects. As a predictor of job preference, the Dollar Metric scale is somewhat better than the multidimensional scale distances. Compare the several indices of performance shown in Tables 3 and 6. The favorable performance of the Dollar Metric procedure is interesting for two reasons. First, it again supports the notion that job preference involves comparative judgments as Soelberg suggests. Second, the scale yields dollar differences between job descriptions and the ideal for each subject. The dollar units are immediately interpretable psychologically as measuring strength of preference, and the dollar differences from a prospective employee's ideal job can be of potential use to employers engaged in searching for personnel. Goodness-of-Fit Measure for the Dollar Metric Scaling Procedure The goodness-of-fit measure between the original preference distances, d'i s, and the Dollar Metric unidimensional preference ijk

I I -23 - Table 5. Dollar Metric Scale Values for Nine Jobs and Derived and Actual Ranks for Subject X. J b Dollar Metric Derived Actual d Job d d Scale Value s Rank Rank 1 - 35'.0 6 6 0 0 2 -230 5 8 8 0 0 3 25. 0 4 4 0 0 4 -213.0 7 7 0 0 5 - 17.5 5 5 0 0 6 150.0 2 2 0 0 7 -256.8 9 9 0 8 236.0 1 0 0 9 92.2 3 3 0 0 d O0 rho 1. 00

-24 - Table 6. Spearman Rhos Based on Dollar Metric Scales for Eighty-Seven Subjects. Subject Rho q1.60 2.87 3.77 4.73 5 40 6.65 7.62 8.55 9 10 -.98 11.7 12 1. 00 13.75 14.83 15.77 1 6.62 17.98 18.18 19.62 20.92 21.87 22.23 23.97 24.55 25.68 26 97 27.77 28.85 29;95 30.83 Subject 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60.90 1.00.83 82.80.87.85.97.57.82.85 -.87.92.97.78.85.88.95.82.98.72.97.92.98.92.83.58.57.92.93 Rho __11 Subject 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 56 (64 pel 75 (86 per 54 (62 per.77 Rho.90.93.87.93.43.70.87.82 85.62.53.85.69.73.63.97.85.92.87.37.95.85.83.63.93.80.97.93 Summary 1 per cent significance (No..78) 5 per cent significance (No. Z. 60) Number of first choices predicted Mean rank correlation > It was impossible to calculate subjects had incomplete data. r cent) - cent) ~ cent) scale values because these

13/ model is referred to as the relative lack of additivity. — The basic component is ( 'ijk the absolute difference between the subjective preference distances, d' s, and the corresponding fitted Dollar Metric ijk distances, 1 sij as in equation (4). ijk (5) j =0 Idj ijk k ijk The total error for subject k is simply the sum of the forty-five "error distances," ijk s, over all pairs of the n = 10 jobs. This ijk sum can be shown as follows: n n (6) (' Z 2. (' k irI > ijk k i1 j> i The d' s may lack the additivity property (i. e., all longer distances ijk A are the sum of the intermediate distances), whereas the 1' s will ijk always be additive. Expression (6) is thus an index of the departure from additivity in the d'. s. Expression (6) would be a comparable measure ijk of fit across subjects if all subjects operated on a preference scale of exactly the same width. Since subjects differ in terms of the width or dispersion of the underlying psychological continuum, 'k is converted to a relative measure which accounts for possible differences in the scale widths. The relative lack of additivity is defined below and is the goodness-of~-fit measure for the Dollar Metric model. ( R (k' = '^R ^ _________ (_7)_ _Edijk /(ni n )/ ) i - 1 j > 13/ Ibid.

-26 -14/ Pessemier and Teach- have developed Monte Carlo expressions r R for approximating the mean and variance of k when the d'.s are.. ij randomly assigned. This distribution is approximately normal and can be used to test the hypothesis that a subject's observed preferences were randomly generated. Relations between Predictability and Error Measures for the Two Scaling Techniques Both the rmultidimtensional and unidimensional scaling of preference involves error. This section will examine two types of error for each method and will relate that error to the predictability of the scaling procedure. The first error measure represents the goodness-of-fit between the original data and the derived model. For the multidimensional scaling this error measure is Kruskal's stress, and for the Dollar Metric scaling procedure the appropriate error measure is the relative lack R of additivity,. Figure 4 plots the degree of rank correlation k. (Spearman rho) between the observed and predicted job ranks from the four -dimensional Euclidean joint spaces and the associated levels of stress. Similarly, Figure 5 shows a plot of the degree of rank correlation between the observed and predicted job ranks and the relative lack of additivity for the Dollar Metric model. In both cases the Spearman rhos were regressed against the goodness-of-fit measure to assess the extent of anear relationship. The R and significance levels the extent of a linear relationship. The R and significance levels are shown below. 14/ Ibid. _

ammawaB a r} 0 eno= 0 * II GEO 0 0 a 0 0 0 0 0 0 0 0 r N t 0 0e CDlC Nrr 0 c lS C) ~ I 0 0 *s 0 0 eo 0 0 0 0 * 0 0 *fl0 o 0 oo 0.0 0 0 0 8D 0. 0 I L 9 I I I I I I I I I ~li~ —~PB B98~i~iarall i C0 0 ~ 0 <0 0 < 0 X. ^4 s p "1.

'AJX-tJPappe Jo 3p1I aA^BelaJT 4surle2 pa44old oqt ueu odsadg '5TI AhIA-TETppv JO lDWGj oAHEI1tEla-J a Ask* *~ 00~ 0~~.. - 0 -0 1* * 0 * * 009 0 *;0 ~e |i * * 0* | * *,* *. 000 Q IB o. 0 ~ 0IA -8?

-29 -2 Significance ~R Level Dollar Metric predictive power (rho) vs. '077.03 Joint space predictive power (rho) vs. stress.131.01 While the extent to which the preference model fits the data (i. e,, size of the error values) is significantly associated with predictive power, one must conclude that the relationship is not important for the ranges of the error measures examined here. Nevertheless, these results are of theoretical interest. The lack of fit between the data and the derived estimates can be the result of several important causes: the subject is a source of pure error, the subject has a low discriminal or preference structure for the stimuli, or the analyst has not properly specified the dimensionality of the point space in the multidimensional model. Neither model lacks the capacity to fit the data or seriously degrades the predictive power because of modest levels of poor fit, Test-retest reliability The second type of error involves the degree to which a given subject can repeat his judgments over time, which is referred to as 15/ the test-retest reliability of the data.- Judgments which remain the 15/ J. Tiffin and E. McCormick, Industrial Psychology (5th ed.; Englewood Cliffs, N. J.: Prentice-Hall, Inc., 1965), p. 129.

-30 -same on repeated trials are said to be reliable. The concern was that unreliable data as defined above might result in models of lowered predictability. An estimate of the test-retest reliability was obtained by having the subjects repeat both the similarity and Dollar Metric judgments a week after they made the original judgments and then calculating the correlation (Pearson r) between the judgments on the two different trials. Only ten judgments were repeated instead of the entire forty-five to lessen the demands on the subjects. Thus reliability can assume any value over the range of the correlation coefficient. Reliable judgments are associated with high positive coefficients and unreliable judgments are indicated by low positive or negative coefficients. In Figure 6, rho is used as a measure of predictive power and plotted against the Pearson r as a measure of retest reliability. The data are similarity judgments used in the multidimensional joint space construction. In Figure 7 rho is plotted against the Pearson r retest reliability for the Dollar Metric unidimensional measures. Table 7 shows a nonparametric median-test of the relation between predictability and reliability 16/ for both the similarity data and Dollar Metric data.- While there is a highly significant relation between the reliability and predictability of Dollar Metric data, there is little relation for the similarities data. It is also of interest that the Dollar Metric judgments were considerably more reliable than the similarity data. Table 8 shows the 16/ Siegel, Nonparametric Statistics for the Behavioral Sciences, pp. 111-16.

r i Rho * Y 1.0..9 5".3.2.1 eb 0 0 olo 0 0 0 0 0 o00@ 0 0 @0.0 + c~~Q * ) 0 0 Q~~~ *b ~ %B~b ~3 ED QB 0 0 0 0 0 0 0* I I 0 0 0 t I I I 1 I I I I I I I ti I l I I I I i r I I ____ac~iP. m_ _ _ _ _ _ _ _ 1.0 -.9 8 -7 -6 -5 - -3 2 -1i..2 3.4.5.6.7.8. Pearson r -.2 Fig. 6. Predictive power (rho) plotted against test-retest reliability (Pearson r) for similarity judgments. 9 1.0 D"""

. I. I. t y Rho 1.0.9.8.7.6.5 w 0 6 *. *, ' * o * '*.0 ~e * * @ 00 * *, * * 0 o *o 0 QD 44 O O ~ oo 0 0 44 a A4 * * L 0 I 0 I I.3.2.1I I I 0 0 Alft —. -- -L I I 'o I I - ~ -1 - -- lkk- lo 44 ~ ~ ~ W i I I I I I I bm ellm I 'I I; I a --- ML — - ---- ---- -.-=...40.10. 9 see -~a -.5-ft. a 4 -2 -, -,3 s:e * I * 2 - I I..4 5.6.. Pearson r * 8 09 10.1.0 a*-4 ' Fig. Predictive power (rho) plotted against test-retest reliability (Pearson r) for Dollar Metric judgments.

Table 7. Nonparametric Median Test on the Relation between Reliability and Predictability for the Dollar Metric and Similarity Judgments. High Low Row Reliability Reliability Totals Test on Dollar Metric Judgments Frequency of Spearman rhos above grand meGrand median dian (. 84) =.84 Frequency of Spearman rhos below grand median (. 84) r.76 Cell A frequency =37 free ra.76 Cell B luency = 4 A + B = 41 C + D = 44 Cell C frequency = 22 Cell D frequency = 22 I I Co 1 Column totals 2 X (df = 1)t' = 14.3, p4.001 Test on Similarity Judgments A + C = 59 B + D = 26 Frequency of Spearman rhos above grand median (.75) Grand median =.75 Frequency of Spearman rhos below grand median (.75) rz. 63 Cell A frequency = 26 Cell C frequency = 13 r., 63 Cell B frequency = 26 Cell D frequency = 22 A + B = 52 C + D = 35 Column totals A + C = 39 B + D = 48 X2 (df = 1) =.92, p<.35 *x2 = [N(1AD-BCl-N/2) / (A+B) (C+D)(A+C)(B+D)J

-34 - Table 8. Summary Statistics on the Coefficients of Dollar Metric and Similarity Judgment Reliability. Dollar Metric Similarity Judgment........ Reliabilit.Reliabilit Mean reliability Range. 76 -34 to 1.00 n Number significant beyond 1 per cent level (r ).76) Number significant beyond 5 per cent level (r. 63) 85 59 (69 per cent) 70 (82 per cent) 54 -. 38 to 1. 00 87 26 (30 per cent) 41 (47 per cent)

-35 - mean reliabilities, ranges, and percentages significant beyond the.01 and. 05 levels. The overall conclusion is that Dollar Metric judgments can apparently be made more reliably than the similarity judgments in this decision context. Furthermore, in practical applications of either model, provisions should be made for quantifying the reliability of the basic judgments since predictions of preference for extremely unreliable subjects are not likely to be accurate, particularly for the Dollar Metric representation of perference. Conclusions and Extensions Quantifying preference for the standard job descriptions by using the concept of an ideal point tended to be more effective when the Dollar Metric procedure rather than the joint space analysis was used. The test-retest reliability of the Dollar Metric judgments was also much higher than the reliability of the similarity judgments. More reliable data on similarities could possibly lead to more predictive joint spaces, even though the relationship between the two was weak for the ranges explored in this study. Distance from an ideal job as a measure of job preference can be applied to computer-assisted man-job matching systems for college graduates. Before visiting a campus companies recruiting college graduates, MSIAs or MBAs for example, might be interested in using the preference data on a set of uniformly described offers similar to those used in this study. Assuming that the raw data from each student were available, a recruiting company might be able to make efficient

-36 -judgments regarding which campuses were likely to have potential candidates interested in the available positions. The underlying concern would be how similar the company's offer(s) is to a given standard(s) and how far the given standard(s) is from the ideal. Objects which are highly similar will tend to be equally preferred, but objects which are 17/ highly dissimilar may or may not be equally preferred. A company could indirectly assess how close any specific offer was to a given student's ideal point by examining how similar its offer was to a standard offer of known distance from the ideal. This would give the firm a prior estimate of how acceptable its offer would be to each candidate. Other sources would have to provide information regarding how acceptable the candidate was to the firm. In an article on computer-aided approaches to employment service placement and counseling, Charles Holt and George Huber note: By programming decision models into the computer which approximate those of the persons involved, the computer can rapidly and efficiently "consider" a great deal of information about many alternative jobs and candidates. In this way interviews can be proposed which hopefully are better than those resulting from manual file searches or machine searches that seek simply acceptable pairings of candidates and vacancies, 18/ 17/ J. R. Taylor, "An Empirical Comparison of Similarity and Preference Judgments in a Unidimensional Context," Working Paper No. 10, Bureau of Business Research, Graduate School of Business Administration, University of Michigan, Ann Arbor, Michigan, 1969. 18/ C. Holt and G. Huber, "A Computer Aided Approach to Employment Service Placement and Counseling," Management Science, XV (July 1969), 573-94.

-37 -We are suggesting that models of the types described in this paper represent significant pieces of information which could be "considered" by the computer in searching for personnel. A firm who wanted to recruit MSIA or MBA students from a particular school might use the following procedure to locate their offer(s) in one or more of the joint spaces of the students. First, the firm would need the vector of paired similarity judgments on a set of uniformly described offers such as the ones used in this study, including the judgments between the standards and the ideal. This similarities vector would be needed for each individual student. These data could be gathered by a college placement office. Since the students usually read a large number of real job descriptions, the standards should not be strange stimuli to the student subjects. Second, the company would need to choose an individual or, better, a group of individuals from among its own employees who demonstrate a capacity to estimate where prospective employees (students) will place new objects in their joint space configurations. This capacity would be discovered by the use of a variety of configurations developed for test purposes. These would contain the jobs used in the standard data on students described in the preceding paragraph and a number of varied additional job descriptions. In other words, the firm would locate an employee(s) who tended to perceive the standard offers and new offers in the same manner as a particular student(s). Third, the firm would present the prospective employee's joint space configurations and its own offer in the standard format to the *-.

-38 -expert judge(s) who had been identified in the manner outlined above. The offer would be located on each dimension of the joint space by the judge(s) and the appropriate ideal-to-offer distance would be computed along with the implied similarity ratings against all standard jobs. Alternatively, expert judgment could be employed by estimating the similarity ratings and then deriving the distances from the ideal job. For example, say there were t standards. The company employee would then make t comparative similarity judgments between the t standards and the company's offer. This vector of t judgments would then be attached to the vector of t(t-l)/2 judgments which had been made by the student. Thus, the company would have a vector of [t(t-l)/2] + t terms, which is the same as the number of pairs for t+l offers. If the vector of [t(t-1)/2] + t terms were multidimensionally scaled, the resulting configuration would be a perceptual space with the company offer located as the t+lst offer. Fourth, if the judge(s) produces configural estimates, the distance of the offer to the ideal can be computed directly. If the similarity estimates are made, the data can be fully rescaled by standard methods or by a simplified approximation using only a limited amount of new similarity data and the old configuration. The rescaling procedure may be used if the configuration of t jobs (the perceptual map) remains invariant when the t + 1st job is added. Configurational invariance

-39 -under changes in stimulus domain has been demonstrated in specific 19/ cases.Figure 8 shows the substantial configurational invariance of t objects (t= 1) after arbitrary similarity judgments have been inserted between all t objects and an additional t + 1st object. The objects numbered 1 through 10 represent the location of the original t standards. The objects numbered 1' through 11' represent the perceptual space after the t + 1st object has been added. The number 11I represents the company offer or new object added to the space. The top t rows (t=10) of the matrix in Table 9 show the input data matrix of similarity judgments which correspond to the original judgments by a student. The eleventh row represents the similarity judgments between the company's offer and the standards. These would have been made by the company employee whose perception was similar to that of the MBA student. In this illustration, the eleventh row was simply an arbitrary row vector whose elements proceeded in numerical order from 2 to 11. Presumably many companies could use the same set of similarity vectors on the standards and place their offer(s) in the perceptual spaces where they could find, within their own company, a person(s) with a similar perceptual vector to that of one or more students. Without making a 19/ Paul E. Green, Arun Maheshwari, and R. Vithala, "Dimensional Interpretation and Configuration Invariance in Multidimensional Scaling: An Empirical Study," Multivariate Behavioral Research, IV (April 1969!, 159-80.

n 71 7 6' 6 I 8' I' 8 0 I a_~~~~~~~~ i I 10 2' 991 4' 5 - ADDED OBJECT \. 2 3 5 II' 4 Fig. 8. Configurational invariance when additional object is rescaled with original set of ten objects.

-41 - Table 9. Input Data Object (Job) 1 2 3 in Matrix Before and After Inserting t the Perceptual Space of t Objects. 4 5 6 7 8 9 + 1st 10 11 2 3 4 5 t jobs 6 6 7 8 9 10 t+lst 11 job 8 7 5 3 5 6 5 3 5 7 4 5 5 9 5 10 5. 7 8 9 6 9 3 7 4 7 6 3 8... I 6 10 2 3 5 10 4 4 8. 4 8 5 4 9 2 4 5 6 7 8 9 5 10 11. * Missing data.

-42 -campus visit, a company could inspect the relation between its offer and the ideal job of students at a particular institution. The above procedure for mapping company offers into joint spaces of students would of course entail additional research. First, there is a question of how readily companies could identify persons within their employ who can product similar perceptual maps to those of the students. Whether preference could accurately be predicted in the new joint space of t+l offers would also have to be explored. Furthermore, the reliability of comparative judgments in a test-retest sense may affect the predictability of joint spaces, although the present research does not provide convincing evidence. Thus a company interested in utilizing multidimensional scaling methods in their recruitment efforts would need reliability data on both the student(s) and the company employee(s) who place the company's offer(s) in the particular perceptual spaces. It would also be necessary to explore the stability of perceptual/ preference relations among the jobs across time. If the perceptual/ preference relations among the standards were not stable over a period of reasonable duration, the procedure described for mapping company offers into particular perceptual spaces would hardly provide useful information. Certainly there is reason to expect that perceptual/ preference relations among the jobs would change as the students were exposed to new influences, not the least of which is the process of searching for a job. If the change was not too rapid, the data could still be timely enough to provide companies with some useful guides for finding personnel.

-43 -While the procedure outlined above is far from operational, the authors feel the: exploratory study reported here serves as a basis for future research regarding application of the multidimensional scaling model to computer-assisted personnel recruitrment,

APPENDIX A Standard Job Descriptions Electronics component manufacturer Production supervision training program —first-line supervisor, 1-2 years; 2nd and 3rd level supervisor, 3-7 years pWork primarily in plants in East: start at Wilmington, Dela. Young management, medium-size firm Salary: $1, 000 per month Nation-wide insurance firm —systems analysis work with computer applications —work at corporate headquarters —first assignment would involve the development of a new companywide system for processing life insurance policies Chicago, Illinois Middle aged to older management Reduced rates on all types of personal and property insurance Opportunity to move to other branches in Midwest as section manager within 3 to 5 years Salary: $950 per month - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -44 -

-45 -APPENDIX A —(Continued) International oil firm Financial and cost analysis and capital budgeting —first year primarily entails "make or buy" analysis Baton Rouge, Louisiana Later possibilities to move into either staff or line management Centralized management Salary: $1,050 per month Very new facilities —new office buildings Computer and software firm, listed on NYSE Technical field sales —normally progress to district sales manager in 5 to 10 years Automobile furnished Santa Monica, California Company recreational and country club is provided free —dining facilities, golf, tennis, swimming, basketball, softball, picnicking, hiking Wide diversity of ages in management Decentralized operation Salary: $990 per month for 1 to 2 years, then straight commission basis ~ — - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

-46 - APPENDIX A — (Continued) Old line industrial manufacturer —industrial gauges, pumps, gears, ballbearings and metal stamped products Marketing analysis and new product market research — chance to move into other functional areas of management later —first project involves responsibility for conducting market potential studies of selected products Houston, Texas, middle aged to older management 500 employees Real estate is very reasonable Salary: $1, 025 per month Small management consulting firm Work on wide variety of problems in financial analysis, marketing analysis, and operations research Special work with Negro-owned businesses in Black Capitalism Program —some teaching of management techniques to groups of clients is available in the Black Capitalism Program Middle aged management —team approach to consulting problems —opportunity to become partner in the firm after 10 years if have the ability. One in five men makes it. Cincinnati, Ohio Salary: $1, 100 per month

-47 -APPENDIX A —(Continued) Work for the City of Denver, Colorado Department of City Planning and Urban Development — department is new and growing Job involves cost analysis and systems simulation —would lead to design of traffic systems and zoning areas in 4 to 5 years —work with variety of urban problems in addition Excellent vacation plan —scenic surroundings Salary: $900 per month, covered by State Civil Service Codes Chemical concern, listed in Fortune's 500 Very selective and intensive management training program —requires trainee who is willing to work hard Trainee is rotated through all functional areas of management for two years and groomed for general management responsibilities Decentralized management structure Philadelphia, Pennsylvania Salary: $1, 125 per month Pharmaceutical firm, 1, 000 employees Production control and production scheduling —work directly with vice president of production operations — would be responsible for new computerized production scheduling system The firm presently has plans to expand into new product area of food processing —will create several high level positions Peoria, Illinois —wide range of management ages Salary: $1, 075 per month

-48 -APPENDIX B Comparative Similarity Judgments Nation-wide insurance firm Systems analysis work with computer applications Work at corporate headquarters —first assignment would involve the development of new companywide system for processing life insurance policies Chicago, Illinois Middle aged to older management Reduced rates on all types of personal and property insurance Opportunity to move to other branches in Midwest as section manager within 3 to 5 years Salary: $950 per month International oil firm Financial and cost analysis and capital budgeting — first year primarily entails i"make or buy" analysis Baton Rouge, Louisiana Later possibilities to move into either staff or line management Centralized management Salary: $1, 050 per month Very new facilities —new office building Please rate how similar you feel these two job offers are on the scale below by circling one number. Very Moderately Moderately Very Similar Similar Dis simlar Dissimilar 1 2 3 4 5 6 7 8 9 10 11 12 13

-49 -APPENDIX B —(Continued) Chemical concern, listed in Fortune's 500 Very selective, and intensive management training program —requires trainee who is willing to work hard Trainee is rotated through all functional areas of management for two years and groomed for general management responsibilities Decentralized management structure Philadelphia, Pennsylvania Salary: $1, 125 per month Your personal concept of the "ideal job" for you. Your ideal job should reasonably be expected to exist. Please rate how similar you feel this job offer is to your concept of your "ideal" job by circling one number on the scale below. Very Moderately Moderately Very Similar Similar Dis similar Dis similar 1 2 3 4 5 6 7 8 9 10 11 12 13

-50 -APPENDIX B —(Continued) Small management consulting firm Work on wide variety of problems in financial analysis, marketing analysis and operations research Special work with Negro-owned businesses in Black Capitalism Program —some teaching of management techniques to groups of clients is available in the Black Capitalism Program Middle aged management —team approach to consulting problems —opportunity to become partner in the firm after 10 years if have the ability. One in five men makes it Cincinnati, Ohio Salary: $1, 100 per month Work for the City of Denver, Colorado Department of City Planning and Urban Development — department is new and growing Job involves cost analysis and systems simulation —would lead to design of traffic systems and zoning areas in 4 to 5 years —work with a variety of urban problems in addition Excellent vacation plan —scenic surroundings Salary: $900 per month, covered by State Civil Service codes Please rate how similar you feel these two job offers are on the scale below by circling one number. Very Moderately Moderately Very Similar Similar Dissimilar Dissimilar 1 2 3 4 5 6 7 8 9 10 11 12 13 * _ w = 0 Al 0_ *S w l ~i~ a-............... |,,S._ 1. r....,_..._~ i.__co*

APPENDIX C Selected Pairs of Jobs as an Illustration of the Dollar Metric Questionnaire Dollar Metric Questionnaire This questionnaire presents, on the following pages, all possible pairs of the standard job offers. Please complete the questionnaire according to the following instructions: (a) Place a check beside the preferred job of each pair, (b) Write at the bottom of the page how many dollars per month would have to be added to the salary of the job you prefer less to make you change your original preference. -51 -

k:; I I I -52 - 9 APPENDIX C —(Continued) First Pair Work for the City of Denver, Colorado Department of City Planning and Urban Development — department is new and growing Job involves cost analysis and systems simulation —would lead to design of traffic systems and zoning areas in 4 to 5 years —work with a variety of urban problems in addition Excellent vacation plan —scenic surroundings Salary: $900 per month, covered by State Civil Service codes Old line industrial manufacturer —industrial gauges, pumps, gears, ball-bearings and metal stamped products Marketing analysis and new product market research — chance to move into other functional areas of management later —first project involves responsibility for conducting market potential studies of selected products Houston, Texas, middle aged to older management 500 employees Real estate is very reasonable Salary: $1, 025 per month

-53 - APPENDIX C- -(Continued) Second Pair Small management consulting firm Work on wide variety of problems in financial analysis, marketing analysis and operations research Special work with Negro-owned business in Black Capitalism Program —some teaching of management techniques to groups of clients is available in the Black Capitalism Program Middle aged management —team approach to consulting problems —opportunity to become partner in the firm after 10 years if have the ability. One in five makes it Cincinnati, Ohio Salary: $1, 100 per month Computer and software firm, listed on NYSE Technical field sales —normally progress to district sales manager in 5 to 10 years Automobile furnished, Santa Monica, California Company recreational and country club is provided free —dining facilities, golf, tennis, swimming, basketball, softball, picnicking, hiking Wide diversity of ages in management Decentralized operation Salary: $990 per month for 1 to 2 years, then straight commission basis

1 6 ll l -54 -APPENDIX C —(Continued) Third Pair Work for the City of Denver, Colorado Department of City Planning and Urban Development — department is new and growing Job involves cost analysis and systems simulation — would lead to design of traffic systems and zoning areas in 4 to 5 years —work with a variety of urban problems in addition Excellent vacation plan —scenic surroundings Salary: $900 per month, covered by State Civil Service codes Chemical concern, listed in Fortune's 500 Very selective, and intensive management training program —requires trainee who is willing to work hard Trainee is rotated through all functional areas of management for two years and groomed for general management responsibilities Decentralized management structure Philadelphia, Pennsylvania Salary: $1,125

-55 - WORKING PAPERS 1. Sidney C. Sufrin, "Reflections on Evolving Competitive Aspects in Major Industries, Oct. 1969, 74 pages. 2. M. Lynn Spruill, "A Scoring System to Aid in Evaluating a Large Number of Proposed Public Works Projects by a Federal Agency," Oct. 1969, 17 pages. 3. John D. Ludlow, "The Delphi Method: A Systems Approach to the Utilization of Experts in Technological and Environmental Forecasting," Mar. 1970, 52 pages. 4. * Claude R. Martin, "What Consumers of Fashion Want to Know," June 1970, 20 pages. 5.^ Sidney C. Sufrin, H. Paul Root, Roger L. Wright, and Fred R. Kaen, "Performance Issues of the Automobile Industry," Sept. 1970, 186 pages. 6. James R. Taylor,. "Management Experience with Applications of Multidimensional Scaling Methods, " Sept. 1970, 27 pages. 7. Daryl Winn, "Profitability and Industry Concentration, Sept. 1970, 102 pages. 8. * Joseph W. Newman and Richard Staelin, "Why Differences in Buying Time? A Multivariate Approach," Oct. 1970, 20 pages. 9. Claude R. Martin, "The Contribution of the Professional Buyer to the Success or Failure of a Store," Oct. 1970, 20 pages.: Copies are no longer available.

-56 - 10. James R. Taylor, "An Empirical Comparison of Similarity and Preference Judgments in a Unidimensional Context," Oct. 1970, 44 pages. 11.* H.. Helmers, "A Marketing Rationale for the Distribution of Automobiles, Nov. 1970, 115 pages. 12. Merwin H. Waterman, "Global Capital Markets," Nov. 1970, 35 pages. 13. Claude R. Martin, "The Theory of Double Jeopardy and Its Contribution to Understanding Consumer Behavior," Nov. 1970, 27 pages. 14. Patricia Braden and H. Paul Root, "A Study of the Sources and Uses of Information in the Development of Minority Enterprise —A Proposal for Research on Entrepreneurship," Dec. 1970, 28 pages. 15. Andrew M. McCosh, "Program Auditing," Dec. 1970, 21 pages. 16. Kenneth 0. Cogger, "Time Series Forecasting Procedures for an Economic Simulation Model," Dec. 1970, 36 pages, 17. James T. Godfrey and W. Allen Spivey, "Models for Cash Flow Estimation in Capital Budgeting," Dec. 1970, 33 pages. 18. W. Allen Spivey, "Optimization in Complex Management Systems," Dec. 1970, 27 pages. 19. Claude R. Martin, "Support for Women's Lib: Management Performance," Jan. 1971, 19 pages. 20. Donald G. Simonson, "Innovations in the Economics of Project Investment," Jan. 1971, 57 pages. 21. Donn C. Achtenberg and William J. Wrobleski, "Corporate Financial Modeling: Systems Analysis in Action," Jan. 1971, 26 pages. Copies are no longer available.

I I l i | i I I II 22. * John D. Ludlow, "Sea Grant Delphi Exercises: Techniques for Utilizing Informed Judgments of a Multidisciplinary Team of Researchers," Jan. 1971, 84 pages. 23. Ross Wilhelm, "The Spanish in Nova Scotia in the XVI Century — A Hint in the Oak Island Treasure Mystery," Feb. 1971, 40 pages. 24. Charles N. Davisson and Herbert F. Taggart, "Financial and Operating Characteristics of Automobile Dealers and the Franchise System," Feb. 1971, 243 pages. 25. H Paul Root, "Market Power, Product Planning, and Motivation, " Feb. 1971, 22 pages. 26. H. Paul Root and Horst Sylvester, "Competition and Consumer Alternatives," Feb. 1971, 21 pages. 27. Dick A. Leabo, "Stepwise Regression Analysis Applied to Regional Economic Research, " Feb. 1971, 20 pages. 28. M J. Karson and W. J. Wrobleski, "Some New Statistical Methods for Analyzing Incomplete Brand Preference Data," Mar. 1971, 25 pages. 29. * Thomas C. Kinnear and James R. Taylor, "Multivariate Methods in Marketing Research: A Further Attempt at Classification," Mar. 1971, 11 pages. 30, W. Allen Spivey, "A Deductive Approach to Linear Optimization," Mar. 1971, 74 pages. 31. Neal Campbell and Cyrus K. Motlagh, "A Stochastic Model of the National Political System in the United States," Mar. 1971, 39 pages. 32. H. Paul Root, "Implementation of Risk Analysis Models for the Management of Product Innovation," Mar. 1971, 62 pages. Copies are no longer available.

-5 8 - 33. William K. Hall, "The Utilization of Linear Optimization Models in Adaptive Planning and Control,"' Mar. 1971, 21 pages. 34. William K. Hall, "The Determination of Optimal Customer Selection and Allocation Policies for Finite Queues in Parallel," Mar. 1971, 22 pages. 35. John D. Ludlow and Patricia L. Braden, "Socioeconomic Development in the Grand Traverse Bay Area," Apr. 1971, 59 pages. 36. Cyrus K. Motlagh, "Evaluation of Input- Output Analysis as a Tool for the Study of Economic Structure of the Grand Traverse Bay Area, "Apr. 1971, 30 pages. *r Copies are no longer available,