Bureau of Business Research Graduate School of Business Administration University of Micliigan October 1, 1969 A SCORING SYSTEM TO AID IN EVALUATING A LARGE NUMBER OF PROPOSED PUBLIC WORKS PROJECTS BY A FEDERAL AGENCY Wiorking Paper No. 2 by M. Lynn Spruill Research'Fellow University of Michigan FOR DISCUSSION PURPOSES ONLY None of this material is to be quoted or reproduced without the express permission of the Bureau of Business Research

\- 3o70 I WI t i I t p OEC 4 1970 BACKGROUND OF THIS PAPER This article is based on research done at the Bureau of Business Research, Graduate'School of Business Administration, University of Michigan, under a grant from the Upper Great Lakes Regional Commission, Washington, D.C.

INTRODUCTION Many times a public agency is responsible for distributing federal funds among a collection of proposed public works projects in order to foster economic development in a region. Although one such agency is associated with nearly every region of the country, the most notable one in recent years has been the Appalachian Commission. Common to all selection problems faced by these agencies is the need to establish a measure of the economic worth of each proposed project. In some optimization problems the worth might be measured in dollars, units of sales, or time. What is the best method of measuring worth has been difficult to determine, though in recent years the use of 1/ "cost-benefit" analysis has received some support in the literature.The objective of benefit-cost analysis is to assess the value of the project from the point of view of the national economy, not merely to estimate the rate of return on the government's investment. Benefits, therefore, are the total increase in the stream of economic values that are expected to result from the costs incurred in the construction and operation of the project throughout its life. Costs, on the other hand, are the expected investment costs and the costs of operation incurred by the government. To compare benefits and costs, it is necessary that they both be placed on the same time basis. This can be done... by comparison of expected annual benefits with expected annual costs. 2/ 1/ C. David Quirin, The Capital Expenditure Decision (Homeward, Illinois: Richard Irwin, Inc., 1967), chap. vii. 2/ Arthur Smithies, The Budgetary Process in the United States (New York: McGraw-Hill Book Company, 1955), p. 337.

-2 - If one considers cost-benefit analysis to be at one end of a spectrum of possible ways of evaluating projects, some thought must be given to tht opposite end of the spectrum: a method which is based almost totally on subjective ideas and in which no justification for the decision is required. It involves farming out the decision to a person thought to be more expert in the particular area under consideration. This expert would then consider the projects on the basis of their written proposals —forms which vary in length from one paragraph to thirty pages for the same type of project — and select those he thought to be best. In this system the agency exercises very little control overthe selection. It is easy to contrast a method such as this with cost-benefit analysis, in which each project is separately and thoroughly analyzed. On practical grounds one cannot expect an agency to perform a costbenefit analysis for all the projects proposed to it. Neither is it logical for the agency to carry out the procedure described in the above paragraph. There must be a middle ground in which the agency can specify the criteria for comparing projects but which will not require the extensive, timeconsuming, and costly process of cost-benefit analysis. This study, therefore,examines an alternative method of evaluating the worth of each proposed project. Developing a system to measure economic -wvorth or to rank investment alternatives becomes, in the long run, just a set of guidelines for the decision maker. These guidelines enable the decision maker to look at two

-3 - investment alternatives and make a rational decision about the relative merits of each. When a basis, or justification, for the decision is needed, the decision maker need only point to the guidelines which have been established. The system of ranking discussed in this paper was developed to assist a particular public agency responsible for distributing a large amount of money to a selected set of public works projects. The contracting agency asked for a method which it could use to reduce, at least to some degree, the subjectivity of its earlier methods of selecting the set of projects to which money should go. The development of this ranking system will be discussed in a series of sections. Each section will begin with the statement of a task as suggested by the Bureau of Business Research at the University of Michigan in a proposal to the contracting agency. Following each statement will be a description of the work and the suggestions that were made to the agency. Standardization The first step of the research proposal was: 1. Identify and describe the projects under fairly standardized headings, so that the full and orderly disclosure of critical informzation pertaining to the selection process is available. A standard format for project proposals was suggested so that the comparable pieces of economic information would be available to the agency

-4 - when the proposed projects were compared. It is believed that such a standard form, attached to each proposal and explicitly stating the relevant economic data, is vital to any objective ranking of projects. It is not practical for any outside group to dictate this standard form. In designing the format, however, the agency should make sure that the proposal will answer specific questions important to the selection process. Two examples of questions which might be asked on this standard form are what change in employment might result if a certain project were undertaken; and, so far as employment is concerned, what portion of the total region under consideration might be affected by the change. It is also apparent that the same standard format for proposals cannot be used with all projects. The questions relevant to the evaluation of a project to improve opportunities for higher education would probably not be helpful in evaluating a project for better health care. If the proposed projects are to be compared only with others of the same class, a separate standard proposal format for each class should be developed. The form for each class of projects should of course include all the questions relevant to the class. However, projects which do not belong to the same class will sometimnes have to be ranked against on0 another because the nlum6ber of d;fferent classes should be kept at a minimum. The following list of classes was proposed to the agency as a workable number.

-5 - 1. Recreation 2. Conservation 3. Highway development 4. Industrial development 5. Health care 6. Education 7. Planning 8. Agriculture Although this list is only a sample,eand the circumstances surrounding each ranking situation will dictate the actual class breakdown, it is believed that most proposed projects will fall into one of these classes. It should be noted that if each of the proposed projects is to be judged against all others, regardless of class, then all proposals should follow the same form. Projects which are in competition for the same amount of money must be judged by the same criteria. In addition to preparing a standard proposal form, the agency must give the proposing person, or group, guidelines to help answer the questions. This can be done by dividing the possible range of answers to each question into several mutually exclusive, collectively exhaustive segments. All responses that fall into one of these segments will have the same rating. This will eliminate the need to maintain a continuous scoring scale and will make it easier for the decision maker to see which projects seem equally deserving with respect to a particular question.

-6 - Criteria The second step of the research proposal was: 2. Establish criteria for the evaluation of the individual projects. One procedure which seems applicable is the identification of six to ten econorric criteria. The criteria that seemed most applicable in judging a set of / proposed projects were discussed with the agency early in this research, since the criteria represent a most important segment of any ranking scheme. Following is the list of criteria and relevant questions which were presented to the agency for discussion. 1. Employment. Will the project increase employment in the region? Does this mean that new workers will be attracted to the region or that the residents of the region may be retrained to handle the new types of work created by the project? In trying to evaluate the employment increase, one must consider not only immediate increases but later increases as the project develops. 2. Per capita income. What will be the relative increase in per capita income in the region affected by the project? Will any increase be caused by an influx of higher-salaried workers from outside the region? Will an increase affect only a narrow segment of the population, or will it affect a broad group? 3. Education. Will the project bring greater educational opportunities for the residents of the region? Will this improvement cause

the residents to leave the region for even better opportunities elsewhere or will they stay? Will people from outside the region - < ' migrate there in order to take advantage of these opportunities? 4. Public services. Will the project improve public services for the residents of the region? Can it be decided whether, for example, a proposed sewage disposal project will provide more public service than a new highway which allows easy access to public parks? Can a proposed project be counted on to provide new public services or will it just improve already existing ones? 5. Housing. Will the proposed project improve the region's private housing? Will it add new housing, or will the improvement consist of rehabilitating old houses? 6. Public investment. Will the project require further public investment as time goes by? If so, is this a liability or an asset? Will the further investment be such as to limit the flexibility of the agency in later years if new and better proposals are made? 7. Private investment. Will the project generate more private investment? How much further investment by the private sector can be expected if this proposal is carried out? What effects will these additional investments have in the region? 8. Tourism. Will the tourist activity in the region be incrcased by the proposed project? Will there be a risk of subjecting existing ment consist of rehabilitating old house s?~~~~~~~~~~~~~~~~~~~C facilities to too much use or will the project provide for new and

-8 - better facilities? Will the residents of the region receive a:';^ fogreater monetary return from the rise in the tourist level? 9. Property values. Will the project increase property valuation in the region? Is it possible that the project will make a region less desirable and hence lower the valuation? Is the expected change in valuation a long-run or short-run change? Will the change in valuation be local or regional? 10. Competing projects. Are there other projects which cannot be started until this one is completed? If so, what is the nature of these projects? Are they sufficiently important to necessitate starting the proposed project? If not, are there "ground floor" projects which should be started in place of the proposed project? It was mentioned earlier that proposed projects may be compared only with others of the same class, in which case a separate list of criteria should be developed for each class. If each project were to be considered against every other project, regardless of class, it would be necessary to use the same set of criteria for all projects. It can be shown that if proposed projects are compared only with projects in the same class the selection decision will hardly ever be optimal in terms of the general criteria. One should realize, however, that an agency responsible for evaluating a large number of projects probably has neither the tirle nor the available talent to consider broad questions about a wide variety of projects. It is possible, and much easier, to compare projects with similar characteristics. The

-7 - loss that occurs in theoretical optimality because of this decision is probably recovered in the saving of time and expense in connection with the ranking procedure. Measurement of intangibles The third step of the research proposal was: 3. Special steps should be taken to collate materials on economic theory with data on the contribution that the various types of development projects make to the long-run economic growth. Such information would be reviewed carefully and used as background by the individuals chosen to judge the proposed economic development projects. A method to account for the differences in the contribution of various classes of economic development projects was introduced to the agency and will be reviewed later in this section. A measure of these differences can be used advantageously by the agency in the ranking procedure, and the development of the measure should therefore be given special emphasis. Like the selection of criteria, the specification of differences in contribution can be reasonably permanent if the agency is careful to define the measure in accordance with available information concerning the long-run economic effect of various classes of projects. The rationale for calculating differences between the long-run economic effects of the project classes is not difficuit to expiain. For example, from past experience in the region it might be detcrmined that more lasting and impressive economic development effect is generated by

-10 - building an industrial plant than by building a hospital. It is not possible for the commission to compare all the effects of these two types of buildings, since many of the effects may be intangible. Nevertheless, one should try to measure the intangible benefits subjectively so that they can be considered in the judging and selection process. It is apparent that no method exists whereby the agency can evaluate all the economic characteristics of a proposed project. Certainly some of the long-run effects of a proposed project cannot be measured exactly. It is also apparent, however, that by careful examination of historical data the agency can make a good, and valid, attempt at measuring the long-run effects on the economic development of a particular region. Perhaps the most reasonable and the most easily implemented method is that the decision-making body in the agency establish its own subjective judgments regarding the differences in long-run economic gains expected for each class or, at the very least, its own subjective ordering of the classes in terms of these differences. It will be necessary to establish a numerical standard specifying the relative difference in the long-run economic effect in each class in order that the. ranking procedure be used most effectively. Scoring The fourth step of the research proposal vas: 4. Develop a quantitative method of scoring economic development projects.

The scoring process can be separated -into three segments. The first segment is determining the differences in long-run economic gains expected from various classes of proposed projects (partially discussed in the previous section of this report); the second segment is weighting the criteria in terms of their importance to the agency in the ranking process; and the -third segment is ranking each proposed project within its class according to each of the criteria. We will consider these three segments singly. There are two reasons for separating proposed projects into different classes. As was mentioned earlier, by grouping similar projects and then scoring them only in relation to others in the same class, it is possible to reduce the conceptual problems in the scoring system. Also, this breakdown into classes will allow the agency to include some of the intangible long-run effects of each class in their considerations. Once the list of classes is determined, the work of the agency should proceed in the following manner. Choose the class of projects which has been judged by the agency to promise the least long-run economic benefit to the region. Assign 1000 points to that class. Now choose the class that has been judged by the agency to be next to the olowest as far as longrun economic benefits are concerned and assign to it a. proportionately larger number of points. For example, if the conservation class of proposed projects was judged by the agency to have the least long-run effect, it would be assigned 1000 points; and if health care projects were chosen as the class next above it, with the difference in their effect estimated at

5 per cent, the health care class would be assigned 1050 points. When each class has been assigned an appropriate number of points the first segment of the scoring procedure is complete. Members of the agency may believe, however, that there are no differences in long-run economic effects of the various classes. In that case each class would receive 1000 points. It is probable, however, that differences do exist, and if the scoring system is to be accurate these differences must be considered. Because the second segment of the scoring system can, and should, be completed independently of the first segment, work on the second segment could be finished before the first is complete. The second segment consists of ranking and weighting the criteria in terms of their importance in judging the economic benefits expected from the class of projects. In this segment of the scoring procedure the points assigned to each class are divided among the criteria; thus, part of the final score assigned to each project will represent the expected performance of the project with respect to each of the criteria. By allocating a certain number of points to each of the criteria, we set an upper limit on the number of points a project may gain if optimum results are foreseen with respect to that criterion. The points should be divided among the criteria in such a way that the number of points a proposed project may gain with respect to each criterion will be an index of how important the criterion is in the judging and selection process.

-13 - What is the most important criterion when the agency judges each class of proposed projects? Is it employment, education, or environmental qualities? The agency must answer this question. Next it should assign a score of 10 units to this criterion. The second most important criterion in each class must then be chosen, and the agency will show its relative importance by assigning to it some score less than, or perhaps equal to, 10. Fractional values of units are allowed in the scoring, and it is certainly reasonable to give the same unit total to several of the criteria if they are thought to be equally important in the judging and selection process. After each of the criteria has been given a unit score it is easy to see how the points given to each class of proposed projects are distributed among the criteria. Simply add the total units assigned to all of the criteria in a class and find what proportion of the total number of units has been assigned to each of the criteria for that class. Let us say, for example, that in a judgment involving the conservation class, employment received 10 units, and education received 7 units; the total of the units assigned to all of the criteria within the class was 40. Employment would then receive 25 per cent of the points assigned to that class, and education would receive 17. 5 per cent. If the conservation class of proposed projects was assigned a total of 1000 points, a proposed project in that class will receive a maximum of 250 points for its expected contribution to employmnent and a maximum of 175 points with respect to its educational benefits. Similarly, if a class of proposed projects has been assigned a total of 1500 points and the same criteria are used, employment benefits would be worth, at most,

-14 - 373 points, and educational benefits, at most, 262. 5 points. Points are assigned to each of the criteria to give the agency a bench mark when it awards points to a project for each of the criteria. If a proposed project is the best in its class with respect to a criterion, that proposed project will receive all the points allocated to that criterion. If the proposed project's expected performance with respect to a criterion is something less than the best, it will receive a smaller number of points. It would be advantageous at this point to recall what has been done on the scoring process thus far. First, each class of proposed projects has been assigned a total possible number of points, as an index of its long-run economic effect compared with other classes. Second, the total possible number of points allotted to each class has been divided into sets of points assigned to each of the criteria. The points assigned to each criterion reflect the relative importance of the criteria in the judging and selection process. A sample of the division of points that could be developed for one class of proposed projects might be helpful; the conservation class has been chosen for the example. CONSERVATION- -1000 points Points Employment 250 Per capita income 250 Public services 125 Private investment 125 Education 175 Dependent projects 75

-15 -The third segment of the scoring procedure was stated to be "ranking each proposed project within its class according to each of the chosen criteria." This procedure can be carried out as follows: 1. Choose the project judged to promise the best performance among all projects within its class with respect to the criterion under consideration. Assign to that project the highest number of points that a project in this class, judged according to this criterion, can receive. These maximum figures were generated in the second segment of the scoring procedure. 2. Rank all the other proposed projects in the class similarly, in terms of their relative performance, assigning to each project a smaller number of points corresponding to the lesser benefits expected from them. 3. Complete this process for each class by project and by criterion. and record the points received by each project. 4. Total the number of points that each project has received and assign this number to the project as a score. This number. represents the relative contribution a proposed project could be expected to make toward the economic development of the region under consideration, according to the criteria that the agency considers to be most important as a basis for judgment and selection. Conclusion What is accomplished by a scoring method that ranks a set of proposed projects on the basis of individual scores? How do these scores assist a selecting agency in choosing a set of projects to be funded? These queJoioiis must still be answered.

-16 - The agency is seeking to optimize the welfare of the region that will be affected by these public works projects. A handy way to do this, given the scoring of the projects, would be to select for implementation those projects which have the highest scores. Except in rare instances this is not the optimal solution, however, because it does not take into consideration the costs of the projects. There are several mathemathical programming techniques which can be helpful in solving the selection problem. The choice of the proper technique depends on the constraints placed on the agency. If the agency is awarded a large sum of money in one installment and is required to commit all this money to the completion of projects, some of which might take ten years or more, the allocation of funds becomes a "knapsack" problem. The common solution to such problems is well known: maximize the total score in each class, subject only to a budgetary constraint. When time becomes an added constraint on the selection procedure more difficult procedures are called for, such as dynamic programming and possibly dynamic Markov Programming Solution. Whether these are available will depend on the complexity of constraints under which the agency must operate. Thus with any method of scoring proposed projects the problems of selection are far from solved even when the scoring is completed. As a part of the research which developed the scoring procedure discussed here, work has begun on an integer programming algorithm to solve certain

-17 - classes of allocation problems, and this attempt will be described later. The main result of the research so far has been to develop a procedure to provide inputs to a further selection procedure. The scores developed are by no means ends in themselves. Some obvious problems come to mind immediately when one considers a scoring method such as the one described in this paper. What if a person cannot tell you exactly what income effect a proposed project will have on the region, but can give only a probability distribution of possible effects? Who is to state what the employment effect of a project will be? Where will the funds for working up reliable statements come from? What loss is taken by judging projects of the same class only against each other and not against all other projects? Many more questions of this type could, and should, be raised. The method proposed here does not answer all questions, nor was it developed with that lofty goal in mind. It does, however, suggest to a federal agency a reasonable set of standards to use when doling out federal funds. It is also an improvement on the decision-making techniques which are being used at present by some of these agencies. The needs of communities and the sums set aside to satisfy them make it important to allocate money wisely. Any effort toward this end is a step in the right direction.

VL6. RECORD # = b1369921. CALL # = WP5300 1969 no.2. TITLE = A SCORING SYSTEM TO AID IN EVALUATING A LARGE NUMBER OF PROPOSED PUBLIC WORKS PROJECTS BY A FEDERAL AGENCY / BY M. LYNN SPRUILL. ALT AUTHOR = Spruill, M. Lynn. ALT AUTHOR = University of Michigan. Bureau of Business Research. I~ tI - ____________________