Research Support University of Michigan Business School November 1998 I/ A META REVIEW OF THE GAME THEORY PUBLICATIONS IN THE FLAGSHIP US BASED JOURNALS WORKING PAPER #98022 BY ARNOLD REISMAN REISMAN & ASSOCIATES ASHOK KUMAR UNIVERSITY OF MICHIGAN BUSINESS SCHOOL AND JAIDEEP MOTWANI GRAND VALLEY STATE UNIVERSITY

I A META REVIEW OF THE GAME THEORY PUBLICATIONS IN THE FLAGSHIP US BASED JOURNALS Arnold Reisman Reisman & Associates Shaker Heights, OH 44122 Ashok Kumar' Visiting Scholar of Operations Management University of Michigan Business School 701 Tappan St., Ann Arbor, MI 48109-1234 Phone: (734) 764-4713 Fax: (734) 936-0279 Email: AshokKumar@ccTMaii.Bt s.Umich.Fdu Jaideep Motwani Seidman School of Business Department of Management Grand Valley State University 310 W. Fulton, #718 Grand Rapids, MI 49504 KEYWORDS: Meta Review, Game Theory, Research Strategies, Application Content Rating. November 3, 1998 Contact Author. Cominents and suggestions on this draft are most welcome.

I ABSTRACT This paper reviews all Game Theory (GT) articles published in the three leading, US-based, OR/MS journals: Operations Research, MIanagement Science, and Intetfaces starting with Vol. 1, No. 1 of each. The articles were first subdivided into Theory or Applications categories. The Applications papers were subclassified on a five point scale ranging from logico-deductive to bona-fide applications. Secondly, the articles were classified in terms of seven types of research processes used by authors. Next, statistical analyses were performed relating data from the above two classifications. The findings show that the OR/MS literature on GT is dominated by articles classified as Theory with no direct real fworld underpinnings and is based on the Ripple process (incrementalism) as' a basic research strategy. The accumulation of theory vs. applications papers has been growing exponentially over time and this is also true of the marginal contributions e.g. those based on the Ripple process. Nevertheless, GT has been found to have a higher percentage of true applications than some other OR/MS subdisciplines. Yet, this percentage is lower for GT than for the total coverage in the above journals. INTRODUCTION The 1994 Nobel Prize in Economics was awarded to recognize the contributions of GT to economic theory. The OR/MS community can justly take pride in this recognition. Operations Research and 3Management Science, respectively published an abstract and an article on GT (GT) by Martin Shubik in their second year of publication (1953 and 1955). Shubik has been a regular contributor eversince: Interfaces, however, published its first article on GT in 1977 in its seventh year of publication (Brams 1977). James'H. Batchelor,(1952) assembled a bibliography on OR/MS in vhich he' included von Neumann and Morgenstem's -(1944) Theory of Games and Economic Behavior. This bibliography was appended to Morse and Kimball's (1951) Methods of Operations Research and listed eighty works about GT, games and gaming (Rider 1992). It was OR/MS that devoted much space in its leading journals during the formative years of GT. Starting 1952, the first 10 years of Operations Research contained 9 GT articles, starting 1954' the first 10 years of Management Science contained 4 unquestionably game theoretic articles, and starting 1971, the first ten years of Interfarces had 2 articles. There were another 7 articles dealing with business or?

I management games in these flagship OR/MS journals. Moreover, OR/MS has hardly abandoned GT in recent years. An electronic search indicated that between 1971 and 1995, 80 GT articles appeared in Management Science, 36 in Operations Research, and 2 in Intefaces. Yet, GT was not very well received among economists in the 1940s. In the immediate postwar period GT was viewed with some suspicion: it was not really economics. It received more attention in other disciplines" (Weintraub 1992). Other than an upstart journal called Econometrica founded in the 1930s by some mathematical economists who "in Herbert Simon's words" - were considered "a sect" (Simon 1959). The mainstream economics journals Nwere not receptive to publishing works on GT in the 40s, 50s and even in the 60s. Both the Nobel committee and the media subsequent to the announcement spoke glowingly about the various applications of GT (Bennett 1994, Peterson 1994, Helm 1994, Pearlstein 1994). It's literature claims many "applications" to a diversity of fields as varied as Economics, Military Science/Var Gaming, Political Science, Marketing, Pricing, Industrial Relations, Negotiation, Bidding, Sports and a broad range of other business problems (Peterson 1994). To which one can add a number of the biological/behavioral sciences, such as "evolutionary competition," "adaptations," c"parental investment in child raising and awhy some animals desert their mates," "animal's fighting behavior," etc. (Pool 1994), and hence the Nobel Prize. As indicated, the literature is very diverse in terms of the disciplines with an active interest in GT. Yet, GT has enjoyed early and continued embracing by the OR/MS community, albeit with some reservations. As will be demonstrated' later the GT literature is ver) voluminous in terms of journal articles and it runs the gamut from theorem-proving to decision-making applications. In this paper) we provide a Is i

I multidimensional profile of that portion-of the GT literature which was recorded in the flagship, US based, OR/MS flagship journals in the period 1952-1995. In the first paper, published in Mlanagement Science (Shubik 1955) described certain terminology related to GT (two-person zero-sum games, non-zero sum games) and provided several potential applications of these types of games, including some in social and organization theory. Typically, the application/modeling papers apply GT to military (combat) applications, market competition (duopoly/oligopoly type markets having two or more competitors), or to one-on-one behavioral issues (bargaining, negotiating, contracting). Although we have not found any formal taxonomy of GT in the literature reviewed, the following classification attributes abound. Zero-sum vs. non-zero sum games, 2-person vs. N-person, dynamic vs. static games, cooperative vs. non-cooperative games, repeated vs. differential games, perfect information vs. imperfect information games, and finally, bounded rationality. In our survey of the OR/MS literature we did not find any papers - specifically dealing with total or unbounded irrationality games. Even though some early OR/MS authors perceived potential problems with GT becoming a managenient tool (Williams 1954; Churchman et al. 1957; Hillier and Lieberman 1967), it has become an integral part of OR/MS graduate programs. In fact, it is so much a part of the prevailing wisdom that because it was not used in research on Barter and Countertrade, a practice that influences close to 50% of world trade, such research was effectively squelched by journal editorial boards (Reisman 1987). There is no question that GT has provided a new platform and'a new paradigm for viewing and for explaining both human and animal behavior having economic, social and/or biological ramifications. But now that it is in its 7th decade (von Neumann 1928), and in its fifth decade within the organized OR/MS communit>y, 'vith hundreds of publications having the sword application in its title, *we raise the questions: What is its 4

I substantive proile within the OR/MS literature? Where in the real world is it really being applied? What are its uses in making decisions that make a difference? Where are the GT implementations that we give the Edelman Prize for? As indicated in Reisman and Kirschnick (1994, 1995) the word application has different meanings to different people. Inasmuch as OR/MS is a subdiscipline of mathematics to some, and a real world problem solving profession to others, this paper attempts to classify the literature of this field along two basic dimensions. One of these: uses a 5-point scale ranging from Pure Theory to bona fide applications (Reisman and Kirschinick'1994). The second uses seven research strategies first discussed in Reisman (1988a and 1992) and applied in Reisman and Kirschnick (1994, 1995, 1996). Obviously, there is a need and relevance of good survey papers, "...two of the top ten most frequently cited articles in Interfaces... were survey papers" (Gupta 1996). In the next section, we outline the definitions of the terminology we have used to classify the GT articles. CLASSIFICATION TERMINOLOGY Definitions of the seven strategies and the article-content classification scale developed by Reisman and Kirschnick are produced below. The research strategies are also graphically depicted in Figure 1. 5

I Definitions, ofPResearch Strategies (Reisman 1988a, Reisman & Kirschnick 1995) Research Strategy Definition Ripple An extension of previous theoretical or applied type of research in a given discipline or subdiscipline. Embedding Bridging Transfer ofTecnolo Transfer of Technology The development of a more generalized formulation or a more global theory by Embedding several known models or theories. The Bridging of known models or of known theories resulting in the growth of the contributing and/or some initially unrelated field of knowledge. The use of what is known in one discipline to model problem domains falling in some other, perhaps even in a disparate, discipline. Creative Application The application directly, not by analogy, of a known methodology to a problem or a research questions that were not previously so addressed. Structuring The process of organization and documentation of the organizational phenomena in the form of models. 'Statistical Modeling Models which arise from analyses performed on empirically obtained data. These models arise from statistical manipulations such as regression or cluster analysis rather than from logical derivations based on various assumptions. The following are some examples of classifications of specific articles on GT under the scheme discussed above. The paper by Harsanyi (1963) furthers the von NeumannMorgenstem theory by adding newa postulates and developing corre.sponding new determinate solutions for bargaining games. Because this constitutes an incremental addition to a well established body of knowledge it was judged to have been based on the Ripple process. The paper by Symons (1968) used the Emnbedding process to meld consumer behavior variables from several disciplines (Xxwage control, taxation of firms and 6

I households, assistance for destitute communities, etc.), in developing models for business and household communities. On the other hand, the paper by Erickson (1990) used empirical data and the Statistical Mlodeling research strategy to show that closed- loop equilibrium better explains dynamic advertising competition than does the open-loop equilibrium. Definition of Article-Content Rating Scale (Reisman and Kirschnick 1994) The literature of OR/MS in general, and that of GT in particular, uses the word Application to imply anything from a bona fide solution of a real world problem, to an interesting model which is but a figment of the authors' imagination. Moreover the word "data" is often used in referring to numbers extracted from a random numbers generator or created artificially to test and/or to demonstrate an algorithm or methodology. Consequently, the articles in this data base were also classified using a scheme developed in Reisman and Kirshnick (1 994). Accordingly, first each paper wvas judged to be either part of the theory literature or the applications literature. Papers falling into the first group are formal constructs which are theoretical in nature. They may be motivated by or even based on real world problems and offer a wide range of potential applications. Yet, the authors have failed to demonstrate their applicability by specific examples. Each theory paper was subclassified to distinguish between those lhich used synthetic numbers for various tests or examples and those that did not. In e it the application of the above scale synthetic data were defined as outputs of random number generators or numbers created by the researcher for purposes of testing or demonstrating a model. If, on the other hand, the paper was judged to fall in the applications area it was then classified on the following five-point scale: I = A figment of the modeler's imagination, a result of logico-deductive reasoning. 2 = A figment of the modeler's imagination that uses synthetic data. 3 = A grounding in the real world 4 = A grounding in the real world with real world data and a demonstrated application that made a difference. 5 = Either category 3 or 4 above with the additional use of synthetic data to test sensitivity, conduct an error anatlysis. and!or explore behavior boundaries. 7

I Additionally, as in Reisman and Kirschnick (1994), within both the Theory and the Applications subsets wve looked for articles which predominantly addressed the following: (1) Meta Research operationally defined as research on GT research and (2) Philosophy/History operationally defined as swork dealing with the historical and/or philosophic aspects of GT research. Each of these sets of classification data were statistically correlated. It should be mentioned if but parenthetically, that in this work we have invoked one.modification to the Reisman and Kirschnick papers cited. Articles which use a previously published scheduling model and proceed to improve the solution technique without adding to the model's real world validation were classified as theory whereas in the earlier study it may have been counted as an application depending on the level of real world grounding contained in the original article. Following are some examples of specific article classifications using the above schema: The papers by Harsanyi (1963) and Symonds (1968) which were discussed in connection with their respective Ripple and Embedding strategy classifications are further classified as belonging to the Theory rith Synthetic Data category as they both shows tables of test results using synthetically generated numbers. On the other hand the Erickson (1990) paper is in the Applications category for using real world data in attacking a real world problem. DATA COLLECTION Initially we thought of using a census approach to classify the entire lifetime body of literature as was done for flowshop scheduling/sequencing research and in cellular manufacturing, respectively (Reisman et. al. 1997a, 1997b). An electronic literature search keyed on "GT" however turned up the following. 8

I ABI/Inform-'a business/management database that indexes approximately 800 journals (no books, conferences etc.). This database purports to include all significant articles in the covered journals. The thiee journals, Management Science, Operations Research, and Interfaces are included among the 800 titles. 1992-August 1994 Disc-a search on the subject GT retrieved 314 articles. 1987-1991 Disc-the same search resulted in 433 articles. AuQust 1971-1986-this search had to be done on-line for lack of a CD-ROM that covers these early years. There were 383 articles with the subject heading GT. The ABI/Inform on-line database covers 1971 to the present. A search in this database may be limited to articles appearing in particular journals. Our search on ABI Inform using GT and related terms as key words yielded the following number of articles in the three journals of interest. * 88 articles on GT appeared in the journal Management Science. ~ 147 articles on'GT appeared in the journal Operations Research. * 5 articles on GT appeared in the journal Interfaces. However, after reviewing the actual articles, we discarded articles that related to areas such as gaming, business and military games, and certain other areas such as competitive bidding, auctions, negotiations, oligopolistic market strategies not employing game theoretic analysis. This reduced set yielded the following number of articles. * 80 articles on GT appeared in the journal Management Science. ~ 36 articles on GT appeared in the journal Operations Research. ~ 2 articles on GT appeared in the journal Interfaces Business Periodicals Index (BPI) - a business/management database similar to ABI/Inform. BPI covers approximately 350 journals (no books etc.). There is much overlap with the ABI database however, BPI may pick tp some articles not covered by ABI. 9

I January 1982-September 1994 Disc - search on GT retrieved 174 articles. Mathematical Sciences (Math Sci) - is described as a comprehensive database of the world's literature on mathematics and its applications in a wide range of disciplines, including operations Research. It does cover journmals, conference proceedings and many books. 1940-1979 Disc - a search on the terms "GT and operations research" retrieved no less than 2513 records. 1080-1987 Disc - same search retrieved another 2978 records. 1988-1992 Disc - same search retrieved 2738 records. WorldCat - this database contains citations to books, journals, films, videos, tapes, discs, musical scores, software - any material that has been catalogued by one of the several thousand libraries around the world including the Library of Congress. There are currently approximately 30 million records in the database. It does not cover individual articles injournals. A search of items classified as books, published in 1940 or later, with the assigned subject of'GT" retrieved as many as 2648 records. This group will include records for dissertations and theses, working papers, as well as for traditional books. However, due to the nature of this database there is much duplication. There.may be several records for the same item because of variations in cataloging. There will also be individual records for each edition of the same title. Initially, we also considered classifying books of readings. A database search of books on GT revealed that Michigan State University alone possessed 287 books in its three libraries (Business School, Engineering School and in Mathematics. Clearly, a census approach would have been very time consuming. Consequently, we focused strictly on all that was published on the subject by the three leading US based OR/MS journals, Opeiatwions Research, Afanagement Science and Interfaces. 10

I — T DISCUSSION In this paper, we provide a statistical life cycle review of the GT literature reported in the premier OR/MS journals. The classification invokes the same categories as those used in the three Reisman and Kirschnick papers cited earlier, with one modification: Articlesv, hich use a previously published game theoretic model and proceed to improve the solution technique without real world validation were classified as theory. This modification was first introduced in Reisman et al. (1997a).. Because this represents a longitudinal review of GT in only three journals, we are able to present the year by year frequencies of articles. The longitudinal review also provides the year by year frequencies of articles falling into each of the categories. Numerically and graphically the year by year additions to the literature (total and by category) are shown in Figures 2 and 3. Tables 1 through 4B summarize the findings. Two of the authors reviewed and classified all 144 articles and resolved any differences of judgment via discussion and compromise. The third author independently. did the same on a random sample. Again any differences were resolved by discussion. It should, however, be pointed out that in a previous study (Reisman et al 1996a) involving the same classifications on another subdiscipline of OR/MS, the authors underwent a thorough calibration of their classification skills as pointed out below. Validation and Consistency Issues in Paper Classifications To check on the issue of consistency of the above subjective ratings/classifications wae took a 10% random sample of the literature (1 7 articles) and asked for two classifications to be made independently of the authors of this paper. One of the outside classifiers had much experience in this process' albeit with a broader Frank Kirschnick is now a Doctoral Candidate at the Engineering Economic Center Stanford University. 11

I OR/MS literature, the other had no experience2. The results are as follows: Cheng's results were identical to ours on 14 out of the 17 articles. In one he differed from us on but one item. Namely, he classified the paper as Application category 4 versus our classification of category 3. In another paper he judged our secondary strategy to be his primary, all else the same. In the third paper he assigned Creative Application as the primary strategy while we judged it to involve the Transfer of Technology. Frank Kirschnick, on the other hand, fully agreed with us in 16 out of the 17 papers in terms of Theory versus Application. The one difference had to do with whether or not synthetic data were indeed used. In terms of research strategies there was complete (both primary'and secondary strategy) agreement in 14 out of the 17 classifications. In one case of the remaining three we concluded that Bridging was primary and Creative Application was secondary whereas he juldged Creative Application to be primary and found no secondary strategy. In the second case we differed only on the secondary strategy. We felt it was Bridging. He judged it to be Transfer of Technology. In the third case our primary Was Creative Application and secondary was Structuring while his primary was Structuring and secondary was Ripple. The fact that both the classification schema and the rating scale were applied in two previously published studies by Reisman and Kirschnick (1994, 1995) on a wide OR/MS literature base, and in another subdiscipline in-depth study by Reisman et al (1996), should not be overlooked as a further quality control check. In addition, it should be mentioned that unbeknownst to the authors until the work was completed and submitted for publication, two Britons, who adopted the same methods of classification as Reisman and Kirschnick and looked at the same issues in their analysis of UK basefdjoumals as well as the EJOR, say the following (Ormerod and Kiossis, 1996): "'To check that we Chun Hung Cheng is a Lecturer in the Department of Systems Engineering and Management at the Chinese University of Hong Kong. 12

I, - understood tle method and its application by the originators we conducted a pilot study. For the pilot study we analysed Interfaces in 1972 and 1992 following the definitions and conventions of the originators of the method. Comparing the results of our analysis with those published we obtained very similar results."' They then proceeded to document the above assertion. Thus we have addressed the subjective rating/classification consistency issue. However, a question which arises almost instinctively is that of validity of the research instrument used e.g., both the scale and the classification schema. This issue will be addressed in discussing each element of the research instrument going from the most transparent element thereof to the one which is the most opaque. The issue of "data" is the most transparent. Authors either presented numbers or not. If they did we looked for the data's source. If there was no indication of obtaining these numbers' directly or indirectly from some real world organization, we concluded that these "data" indeed were synthetic. In this regard there was rarely any question among the. evaluators. While in this process we also looked for any real world grounding/motivation specific to the research being reported on. If none were to be found we concluded that it was Theory irrespective of any application claims by the author(s). Work with or in Company X or institution/organization Y was judged as an Application. Here, however, we were sensitive to any disguises that might have been invoked, hence in a few instances it was a judgment call. Philosophyl/History articles by their nature were easy to pinpoint. For example, this very paper has elements of both the philosophy/history and the Meta Research categories albeit it is basically Mieta Research. In classifying the research strategy used the Ripple process was the easiest to recognize. In this respect the authors themselves were vers helpful. There was much prior literature to be cited and it all was directly relevant to the problem being addressed. 13

I On the other'hand, the Structrring process was easy to identify for opposite reasons. There was a paucity 6f directly relevant analytical or OR/MS type literature to be cited. Instead the authors typically cited descriptive, institutional or contextual literature. The Creative Application process too was relatively easy to spot as the author clearly applied a well established OR/MS methodology to a problem not previously so addressed. In the use of the Etbedding process the authors were helpful in telling us that they have created a model etc., that is more general then its constituent parts obtained from previously published literature. When Transfer of Technology was invoked here too the authors were helpful. They often said as much and the publications they cited were from other, often disparate, disciplines or model domains. The Bridging process was the least transparent to identify as it was sometimes, on first reading, confused with Embedding or the Transfer of Technology. The Statistical Modeling process is easy to identify, as indicated in an earlier citation of the Erickson (1990) paper. It should be mentioned that at times the authors used more than one of the above strategies. In these cases we have identified the primary and a secondary strategy. Criteria for Model Validation The issue of model/theory validation has been of concern for many generations in the general literature of science. This concern has not escaped the literature of OR/MS. In fact, in 1993 the European Journal of Operational Research dedicated a special issue to this subject. Specifically, Little (1970), Roberts (1977), Eilon (1979), Powers et al. (1983), and Gratreick (1983), among others, argue that validation should be based on the criteria of simnplicity; transparenc andfleribility Landry et al.(1983), and Banville (1990) argue that a valid model is a legitimnate model. While, Toulmin et al. (1979) and %McCloskev (19S5), argue that in ORMNIS a convincing argument may serve for model validation. And there are allways "the coi-respondence criteria that measure the degree of 14

I -c conformity of the model to empirical facts" "put forward by such OR pioneers as Rowe, Williams, Blackett, Waddington, Morse, Kimball and Koopman." (Dery et al., 1993). In the quality control discussion of the previous section we have shown that our methodology is simple and transparent to the point in some aspects that it is intuitively obvious. It is flexible in that it has been applied to other, rather different literatures, and by other investigators. Its legitimacy is addressed by several of what we consider to be convincing airgtuments. Lastly, it corresponds to the views, based on a Awealth of empirical observation and reflection by many very seasoned contributors to OR/MS theory, practice and education e.g., Lillien, (1985), Ackoff (1987), Blumstein (1987), Miser (1987), Geoffrion (1 992); to the findings of other contributors to Meta Research on OR/MS, Corbett and Van Wassenhove (1993), as well as those of an impartial scholar of professions in general (Abbott, 1988). RESULTS It is interesting to note from Table I that of the 144 papers identified to be GT in the leading OR/MS journals, the bulk, 69, or 47.9% of the research is based primarily on the Ripple strategy. Another 12 papers or 8.3% used the Ripple strategy in a secondary role. Thus, 56.2% of the papers used this strategy in either primary or secondary category. Alternately, of the 81. papers that invoked the Ripple strategy 85.2% used it in a primary mode and 14.8% in the secondary mode. Moreover, the. 144 papers yielded a total of 182 classifications. Of these the Ripple strategy accounted for 44.5%. Also, 34.6% of the total classifications fell into the Creative Applications. Br-idging accounted for 10.4%. Statistical Modeling 3.8%. Embecldding for 3.3% Structuring and the Transfer of Technology accounted for 1.6% each. The fact that only one paper in the entire flagship US based OR/NIS literature on GT used Structuring as a primary research methodology is most interesting. To be sure, the C-erieave Application strategv wNas invoked as a primary 15

I process in 55 or 38.2% of the papers and another 8 or 5.6% used it as a secondary strategy. The apparent reason for such high incidence of use of the Creative Application strategy was that these articles applied GT.concepts to model or solve an ';application". Finally, 38 out of 144 (26.4%) papers had some secondary classification, i.e., these papers employed a dual research strategy. Table 2 on the other hand, shows that 52.8% of the papers concentrated on extending theory, of which 25 or 17.4% did do some testing using synthetic "data" numbers, whereas another 45 or 31.3% did not use any numerical tests. At the same time 47.2% of the papers fell in one of the various categories of Application. Of the 68 papers. on the Application side 38 fell in categories 1 and 2 which means they did not even attempt to have any real world grounding, and none dealt with Meta Research. When we combine these 38 Application papers with the 76 unquestionably Theory papers we arrive at the fact that 114/144 or 79.2% of this literature did not reveal any direct grounding in the real world. At the other extreme 30 of the 144 papers (20.8%) fell in categories 3, 4 or 5. Thus during the period 1954-1995, on the average, less than one paper per year (3 in four years) grounded in the real world appeared in the leading OR/MS journals. It is interesting to note that contributions to categories I and 2 of Applications columns are fairly uniformly distributed over time starting 1955, while the contributions to Pure Theory as such, did not begin until 1967. On the other hand after the Haywood Jr. (1954) paper classified as Category 5 (real world) Application, no such paper appeared until 1978 (Billera et al. 1978). On the introspection front, only 4 papers concerned themselves primarily awith AMeta Research and only 2 with the philosophy of this field. Interestingly enough all this introspection occurred on the Theory side as shown in Table 2. 16

I Combining these two independent classifications we find the following significant observations: (1) A subset of 33 papers (22.9%) of the literature can be described as having dealt with Pure Theory while invoking the Ripple process as a primary research stratagem, (2) A subset of 17 papers (11.8%) accounted for simultaneous classification in Theory with Synthetic Data and Ripple strategy. In all, therefore, a large subset of 50 papers or 34.7% of all the papers classified were Theory (not including Meta Research, Philosophy/History) and Ripple combination. (3) Another subset of 9 papers or 6.3% fell into Application categories 1 and 2 (e.g., no real Nworld underpinning) while relying primarily on the Ripple strategy. Thus, 41% of this literature can be described as making marginal (Ripple) contributions to the theoretical aspects of GT. TREND ANALYSIS The most interesting findings appear in Figures 2 and 3, Tables 3A and.B, and in Tables 4A and B. These tables present the mix of papers published in 1952-1961 (first ten years) and in 1986-95 (last ten years) and the first and last five year periods respectively. They also present the percentages corresponding to those shown in Tables 1 and 2. Specifically, the largest subset in the first ten years in terms of research strategy is comprised of the Creative Application 'strategy. However, we note as a primary strategy this fell from 58.3% to 43.4% from-the first decade to the last. We see a similar trend in the Embedding strategy. In the first ten years, there were 8.3% of papers classified a's using Embeddino but in the last ten years there were only 3.8% of papers classified as such. However, the trend was reversed in the case of Ripple as primary strategv which increased from 25% to 35.8% between the first decade and the last. Similar observations apply to the first and last fie-ear data for Cerive lionand Ripple astrategy. The Creative Application strategy wvas used by a lesser percentage of papers in the last five years as opposed to the first five years. On the other hand, 36% of the articles used 17

primarily the Ripple strategy in the last five years whereas 33% used it in the first five years. In terms of the Theory/application classification, the percentage of papers primarily addressing Theory increased from 30.8% to 38.2% from the first decade to the last, although there was an increase, from 24.9% to 30.1% in the application papers which discussed bona fide real world studies e.g., categories 3, 4, and 5. Perhaps the most remarkable observations from Tables 3A and B are the following: From the first decade to the last, the percentage of papers primarily classified as Pure Theory rose from.0 to 24.5%. Parallel data for first five years to the last five years indicates that Pure Theory papers rose from 0 to 32%. Even if we recognize that the sample size for first five and ten years is small, this increase in theory papers is significant (see Figure 3). Interestingly, there were no Embedding, Bridging, Transfer of Technology, Structuring processes nor Statistical Modeling users reporting in the first 5 years of this literature. In contrast, only the Transfer of Technology process was unreported in the last 5 years. Finally, articles using Structurring as a primary strategy were never reported on. Figures 2 and 3 numerically and graphically display the year by year production and accumulation of articles published by Operations Research, Management Science and Interfaces in this field. Starting 1971 (see Figure 3) the accumulation of theoretical papers clearly and irrevocably outpaces the publications concerned with applications real or imaginary. Coincidentally, this is also the year when the use of the Ripple process (Figure 2) by far starts to outpace all the other research strategies combined. When we extract the Categories 1 and 2 Applications and lump them in with the Theory subset wbe find that over time there were indeed very few applications that were directly grounded in the real world. As indicated earlier there were only 30 papers (20.8%) deemed to fall in Applications categories 3, 4 and 5. On the average. this translates into 3 such papers every four years, i.e., less than one paper of real-world application per year xithin the entire lifetimes of the three US based flagship OR/MS 18

journals! Although this indeed is a higher percentage and/or frequency than what we have found to be the case for the life-cycle (all articles in all journals) of some other, albeit younger, OR/IMS subdisciplines eg. Flowhop Scheduling and Sequencing (Reisman et al, 1966a) and in Cellular Manufacturing (Reisman et al, 1966b) it is much less than the three journals surveyed for this paper have published for all of OR/MS. In fact the corresponding percentages (based on page counts) were 30% in 1962 and 32 % in 1992 (Reisman and Kirschnick 1994). This raises some profound questions. Is GT not very useful. in the practice of OR/MS? If such is the case why does it play so important a role in our graduate curricula? If indeed it is, then why has such practice not found its way into the record of our flagship journals in a more proportionate manner? CONCLUDING REMARKS There is no question.that GT has made major contributions in reorienting the way we think about and explain behavior in a large variety of interactive settings. These. settings involve humans and/or animals. There is also no question that the Nobel Prize committee made a good choice. This paper however, set out to examine GT from two perspectives. Seven decades after its founding by von Neumann, being an integral part of graduate education in OR/MS, and having a worldwide literature base countable in the thousands of books and articles, is it indeed being used like linear programming? Is it' being applied to make strategic and/or logistical decisions? Based on our findings resulting from the content analysis of the flagship OR/MS journals, we conclude that at best it is basically a stimulus to a *way of thinking. It is not yet an item in the decision maker's tool box and its advances in the literature surey)ed tend to be rather marginal and theoretical. 19

I I. Yet, it-does have an appeal to the more mathematically inclined. Now especially after the Nobel recognition, it is highly prized in academic circles across several disciplines. To the extent that OR/MS is directed at sblving problems of consequence, to society, to institutions or to corporations, it must guard against the temptations to force-fit game theoretical approaches where others would be more realistic and/or implementable. In addition to providing a quantitative (statistical) review of what has been accomplished, this kind of exhaustive and systematic classification of research in a given field makes the voids in that literature very visible (Reisman 1988b, 1992). Thus, one can logically suggest that journal editorial boards should be most receptive to papers documenting true (real world) applications of any of the GT models. Based on experience in other realms of OR/MS research (Reisman and Kirschnick 1994 and 1995), and in the findings cited earlier for this research, it appears that invoking to a greater extent, research strategies such as Transfer of Technology, Bridging, Embedding, Creative Application, and Statistical Modeling would provide a higher probability of a breakthrough in both the theoretical side and the applied side of the field and on further contributions to Meta Research. Lastly, Dery et al. (1993) point out that'... as can be seen, the model validation issue cannot be taken for granted. It even is a topic of debates in OR." The authors of this paper will be the first to admit that we may not have taken all possible measures to validate both our classification scheme and.r our data collection. We have, however, done more in this regard then the bulk ofthe extant ORGU-S literature in general and that of GT in particular. 20

- ~-ry REFERENCES Abbott, A. 1988. The System of Professions: An Essay on the Division of Expert Labor. University of Chicago Press, Chicago, p. 1 1 8-121. Ackoff, R. 1987. "OR: A Post Mortem," Operations Research, Vol. 35(3), p. 471-474. Banville, C. 1990. "Legitimacy and Cognitive Mapping: The Study of Social Dimension of Organizational Information Systems" Ph.D. Dissertation, Laval University, Quebec City, Canada. Batchelor, J.H. 1952. Operations Research: A Preliminary Annotated Bibliography. Clevpeland: Case Institute of Technology. Bennett, A. 1994. "Nobel in Economics is Awarded to Three For Pioneering Work in GT." gWall Street Journal. October 12, p. B 12. Billera, L.J., D.C. Heath, and J. Raanan, 1978. "Internal Telephone Billing Rates- A Novel Application of Non-Atomic game Theory," Operational Research, Vol. 66(26), p. 957-965. Blumstein, A. 1987. "The Current Missionary Role of OR/MS". Opderations Research, Vol. 35, p.926-929. Brams, S.J. 1977. '"The Netwrok Television Game: There Bay be No best Schedule," Intefaces, Vol. 7(4), p. 102-109. Churchman, CW., R.L. Ackoff and E.L. Amoff. 1957. Introduction to Operations Research. New York: Wiley & Sons, Inc. Corbett, C.J. and L.N. Van Wassenhove, 1993. "The Natural Drift: What Happened to Operations Research?," Operations Research, Vol. 41, p. 625-640. Dery, R. 1993. "Revisiting the Issue of Model Validation in OR: An Epistemologic View," European Journal of Operational Research, Vol. 66 (2), p. 168-183. Eilon, S. 1979. Aspects of Management, Pergamon Press, Oxford, UK. Erickson, G.M. 1990. "Empirical Analysis of Closed-Loop Duopoly Adveriising Strategies.":fctangemnter Science, Vol. 38(12), p. 1732-1 749. Geoffrion, A.M. 1992. "Forces, Trends, and Opportunities in MIS/OR." Operations Research. Vol. 40, p.423-445. 21

I Gratwick, J. 1983. "The Importance of Being Trivial," Interfaces, Vol. 13, p. 59-61. Gupta, U.G. 1996. "Using Citation Analysis To Explore The Intellectual Base, Knowledge Dissemination and Research Impact of Interfaces (1970-1992)," Forthcoming in Interfaces. Harsanyi, J.O. 1963. "Rationality Postulates For Bargaining Solutions in Cooperative and in Non Cooperative Games," Management Science, Vol. 9, p. 141-153. Haywood Jr., O.G. 1954. Military Weapons and Game Theory," Operational Research, Vol; 2, (4), 1954 p. 365-385. Helm, L. 1994. What Are They Doing, Playing Games? Los Angeles Times. October 19, p. B.. Hillier, F.S. and G.J. Lieberman. 1967. Introduction to Operations Research. San Francisco: Holden Day Publishers. Landry, M., J.L.Malouin, and M. Oral. 1983. "Model Validation in OR," European Journal of Operational Research, Vol. 14, p.207-220. Lillien, G. 1985. "OR/MS On Thin Ice," Interfaces, Vol. 15, (4), p.12-13. Little, J.C. 1970. 4"Models and Managers: The Concept of a Decision Calculus," Management Science, B 16, p.466-485. McCloskey, D.N. 1985. The Rhetoric of Economics, The University of Wisconsin Press, Madison,. WI.. Miser, H. 1987. "Science and Professionalism in Operations Research," Operations Research, Vol. 35, p.314-319, 1987. Morse, P.M. and G.E. Kimball. 1951. Methods of Operations Research. rev. ed. Cambridge, MA: MIT Press. Ormerod, R. and I. Kiossis, 1996. "OR/MS Publications: Extension of the Analysis of the US Flagship Journals to the UK. Forthcoming in Operations Research. Pearlstein, S. 1994. TWhat's at Play in the Mlarkets: Economists Win Nobel for Gauging GT's Business Role." The JW'ashington Post. October 12, p. FI.

I Peterson, J. 1-94. "Two Americans, German Win Nobel in Economics." Los Angeles Times. October 12, p. Al. Pool, R. 1994. "Economics: GT's Winning Hands." Science. 266, p. 371. Powers, R.F., J.J. Karrenbaner and G. Doolitle, 1983. "The Myth of the Simple Model" Interfaces, Vol 13, p. 89-91. Reisman, A. 1987. "Where Have We Lost Our Way?," OR/MS Today. October, p. 7. Reisman, A. 1988a. "On Alternative Strategies for Doing Research in the Management and Social Sciences," IEEE Transactions on Engineering Management. 35(4), 215-220. Reisman, A. 1988b. "Finding Researchable Topics Via a Taxonomy of a Field of Knowledge," Operations Research Letters. 7(6), 295-301. Reisman, A. 1992. Management Science Knowledge: Its Creation, Generalization and Consolidation. Quorum Books, Westport, CT. Reisman, A. and F. Kirschnick. 1994. "The Devolution of OR/MS: Implications From a Statistical Content Analysis of Papers in Flagship Journals," Operations Research. 42(4), 577-588. Reisman, A. and F. Kirschnick. 1995. "Research Strategies Used by OR/MS Workers as Shown by an Analysis of Papers in Flagship Journals," Forthcoming in Operations Research, Vol. 43(5), p. 731-740. Reisman, A. and F. Kirschnick. 1996. "Statistical Discrepancies Between What is Used in the Field and What is Published in Flagship OR/MS Journals, Working paper. Reisman, A., A. Kumar and J. Motwani. 1997a. "Flowshop Scheduling/Sequencing Research: A Statistical Review of the Literature 1952-1994," IEEE Transactions on Engineering Management, Vol. 44 (3), August, p. 316-329. Reisman, A., A. Kumar, J. Motwani and C.H. Cheng. 1997b. "Cellular Manufacturing: A Statistical Review of the Literature: 1965-1995," Operations Research, Vol. 45(4), p. 508-520. Rider, R.E. 1992. "Operations Research and GT: Early Connections," in Toward a History ofGT, E.R. Weintraub (ed.). Durham, NC: Duke University Press. 23

I Roberts, E.B. -1977. "Strategies for Effective Implementation of Complex Corporate - Models," Interfaces, Vol. 8, p. 26-33. Symonds, G.H., 1968- "A Study of Consumer Behavior by Use of Competitive Business Games," Management Science, Vol. 14 (7), p. 473-485. Shubik, M. 1953. "GT and Operations Research/' (abstract). Journal of the Operations Research Society ofAmerica. Vol. 1(3), p. 152. Shubik, M. 1955. "'Uses of GT in Management Science," Management Science, Vol. 2(1), p. 40-54. Simon, H.,1959. "'Review of Elements of Mathematical Biology," Econometrica. Vol. 27(3), p. 493-495. Toulmin, S., R. Rieke and A. Janick. 1979. An Introduction to Reasoning, Macmillan, New York. von Neumann, J., 1928. Zur Theorie der Gesellschaftsspiele 1959. Mathematische Annalen. 100, 295-320. Translated by S. Bargmann as On the Theory of Games of Strategy. In Contributions to the Theory of Games 4. A.W. Tucker and R.D. Luce (Eds.). Annals of Mathematical Studies. 40. Princeton NJ, Princeton University Press. von Neumann, J. and 0. Morgenstem. 1944. The Theory of Games and Economic Behavior. Princeton NJ, Princeton University Press. Weintraub, E. R. (ed.), 1992. Toward a History of GT. Durham, N.C., Duke University Press. Williams, J.D. 1954. The Compleat Strategyst, Being a Primer on the Theory of Games and Strategy. New York: McGraw-Hill Publishers. 24

Trale' 1?- cin'ssinvottann trc-nitlltenry IPapcr Putflldirl In Monalocnr~int 5Sclenee, Opervilnnm flmrardel u1 In 1nirrfat-c Twimi tin. wri I C b4 Ifoio 9.d.. r -h. r 4,vA Ve A PPid. SIAlvolhh.1 Y~r *~3~' i- a.~ a. 'a I'2 un I____ V__________ I '86.1 If I~~~~~~~~~~~~~~~~I INA 4. I J78. 3 VMS7 S 1'1, 73 I ______ VW__________ 41 I V 9 9 5 1 8)43 ~~~~~~~443 8'tal 4.3~~~~~~~~ 35 LI ' 08, 5I 3, O 7 ~ ~~~~~~~~~~~~~~~~~~~~ J 4 3i I'llonaly.Strala-ty 11 NrcoodarySiva(va

Table 2: Lihissdfic1tIln 1 Mh1 irc'um icor'I~y I'-.pers Publihhcd fit Mmungerent5clence, Opera3tions Resenrch, siid lIn hlterriecN lus'd on Thea,/Appi~catimlti C.-ategnrles I I I T j 11114. * 61..... Ittstwv " I hk440.Ilc U44 I -Iti.lay -- F F: F F: F ll- " i IP I i F7 P: FZ T --— T I 45 4~~~~~~2I 331 3A 3114 2 3 IJ W3% 39/4 3 39711 39/ IO 1993 1~~~~~~33 ___ Yor 11W 11w 1 or2El 393(4a~of 4 diet3 2II~I3 IC 39301 W.I 1/& 2 d____I 3 1 _ 7_2____ 391133.3~~~~Wlt " 1s 21.hll - 3 '4)3') ~ ~ ~ 1- I3.3..rhwh~3r~.g3(.93 "4-.......', 1 '-.. * *..... _ _ _ _ _ _ _ _ _ _ _ _,r

(1V-aveu on Rewuerci Slrntcgy) lt1 Hipfe I., raahedJ~nc2.: j I1$4dgIng '. jrnni..r'0rch.. '. jcrevatv Applic. stru-clufIlg N~tL oddIi.oc 'rtai Nei 12va 3 it,171 0 I MLI CU~~lrlchinn (P o mX)(1rorOny jT Cj0I, _ _ _ _~~~~~~~~~~~~~~~~~~~~~~. 4.;D'. 5.3..' 7 1Rich Clamnifkution (Ilisrud on 14 Puaper-s) ~ Pit' ny,JFktI Ton IJoh~r )iCwhl Xtr~aw'y (P 4- S) (ro.~rna~ ________ I focctq r a?(AwhAsiftrlcuinnvi Io 19',. 21 4 ' 7.1 71.10 043 (10 1. 1nhC~cate(' rrpr (ibti'., nlit 16 11PImpr) ('i~r+cod,)________ Ti;1No. f Puervt, 19. 4 * 1 53 Ends C'1hrit-1flrion (P or S) (Prdoury Only)19i u. z z 1'L'rct'u~t:Ike ofloculNo. of Pipers to 100'/3S. 9 IS.3 H. 1.9..94. 75 1) 1). 4.4 3H 00 1.. I'raci Cal N tifedcuion e sil tions 5 611ers '"r '1 3 9 I 95 1 6. (10t1T I~nakr 1eatch Sl ruut cgy O1' 4~ S) (rmr~eni RiMch (Xwguory (the%ed (oi 6 11"Piprs). 1ret+~odaery).A~~.' 4 1 ~ ".~, ~,4, 1061) I1Yei5) l rimu.n'Sraccply -. Secontlarystlrtcgy

Ta;ble 313: 5-Yeorl~'rvid An21v~ic orGamTn h ~'eory Popcrs Publisic-d In Managment Scence,Opteraf(1nn Rescorch. -and in Tuuierracev (flused un Rewcrch Strsttgy) I -- — y -~ ~ ~ -- -v I. - ti~ppic __ __ld~t... I IItrdgn.2!fl I I'mint. of'rech.-... lCrtative Appllc.INfracturbig MI All.iauelireq IP' I ' t A P. I 5-;- j jP..5 I..I s -i 31) 9 P) (TuAIN4 f'pr 1 0 ( 0 (b 2 0 ( 1 ( ('-nswat~ i A~. a~rP,;pc, if,(IV 3.3 3 )( (0 00 1 0 0.0 0.0 66.17 0. 0.0 (LB 0. (.0~ _______ _______ _______ _______ _______ __n___ (P rim ary O nly)........ ~. rn412 Nit. n(lw(A firart~l 002 0(95 - UndetLr 1Eii Strraw P* )(rmr+S odi) ___________________ _________ __________ _________ ________ 1~~*(u~e ~r C r~%Ir~~eaui4In~ i.~ 100. J333 (1.0 0.0.0. 46.7 (1.(0. T1n01 Nul. u(I1oapc.r-; ill 0~ 3 1 - ~. 1-'sc-rliiti3i 4m %.0N 3 bho. ii133.363 1'( J. U' 16.0 12() (.M.01 36.0 0(3 11.(Ij4.0 9.0 Total Nil. or ('i Icair.34 1'.. (9 ('or~~rntc~~I. of (:l~~ilk~~cion~ Jo.. 10(1% -.,~~~~ ~.2.9. 2.6, (L. 1, 2.. 5 Prnziy e(ati' c 3 Z~occtitic ary S4 rR4t.Ny Yean 1956) 3995)o ' P - l1rimar), Ntr*((oxy - S-SecondarY.Ntra(ev

'!'uhle -f.tAI fl-Yc:tr Trend Analyxds nrmnne Theory Pnpers Publihecd in Mannigcmcnt Science, Operations RXL'areLT2, anrd in Tn(erfnccn (Unsci! (inl Theory/Applicaf ion Catcgorl'Mtatinf).. — ' — A~ppi~calonK. *.' N 1. k., -.:* - 1it4tN.1 Al(Popnmb I.l 53, I ~.2. ~:. '.6.$ O.. 0 0 1J 13 0 5 0 1 0 0 vkids (Thsetitmlcsln (P r4 l (rrI~r Ony' *,;0t,Ptc.I ruiyratnect. IiowI No.,Wrtpwrusa.4 118... 22.6 1.3 44 1 1.3.91.4 00.0.0 0.0 0.0, 243 0.0 94 0.0 29 0.0 U' f t.(:(%1wt1k1iwn(11AvM o i P5 jw:.P7nI~i ~ 12'.__ _ _ _ _ _ _... 1 1 ZIIls e Puirrfilackssrs(.1etgy 11tr ). Plia rmndP:~ '?$... 4tb.tct5silvR%; __________ - 0% ~~Q: I:~.{.>S;~s..., T T'n -ycu oln~e(Cbuae 1I1cvIe. lif A) Ppwc~ite lon.. & -4v:;'t4shIA vli..e~N~. $ n. 44H * ' P1'rlmatry-Slntcjgy - S. Secondnry Slirkiey II

'fable 4ll: S-YertilrcndAnuayld fCawTerPprsubihdnMngmntScicece, Operatlons Reseachtl, and in Interraces (flased on Tlcoi-ylApplication Cntegorfzaf ion) I.. -... 4. I.,.1A nnlkut~lonnn-... -....*, I 'Jis Teory -- L r - r — "'"'-r'r -r 't PI;lc pty I -.4~~~~~~~~~~~~~~~~~ KIC1 C~ilinrfla (P o nS Pser frimary OhlyW ) ' i r14' __ Porcrr;hfg r~o(1;;(MNoof Paper% lu, OT... 0 13 0.0.3 333 0.0 00 0 0 0 0 0 0 0.0 110 3 Tuilal Nil. ofi ClvfIn, ifltiatlrs a' U. r ~' 3; 4 I tid rhAc Strum~r. P+{ (Prlms',+SecoAutsAi1- ~- EsriCateory flasd opeff Parperv)... Q n+vud~ i.. o _ _ _ _ - -joA1dx~~ 41nr 1,1;'..rr.n a.....ft'hniflflloIln. fit A m..b.rm;bnI 'I 0m,....54. 2....4.. I i I., I VI WC III A KV -1-1 II 'VW If M I I III II 13 III LL8141111tr In 7;.. '... I. t. -.-.,. 4; I ';',,.. I I.. -.,-.,.". '.,,. -., -:.......I 1., -.I r;, - - '. '... I -- ".. W.,.I 'i. I. t..1.1. -;..... - "', -.,. I s.'.0... I. I,..,. 41. 1. ":;,:,;, 1; i 'I, -, - %. '[. I.. A rnlff-?fh&-nvW: 1.. -,. ", -. I,-., '.:r -,.,."I ",, I i-j 1..1 I' II dN;;V. ';grvic 1(I.3.C-0 4 IQ30,!rMa 2 0 41 a 0,1! FEXr C'6rsi0Clflc tI; (O or 8) 4mlr 01 __L2PimoiI~zjjj~4-j. 0.4..4) Prc ut; r uf';;r No.; l'pcs1;,v100/.o U...1.0 12,0. 0.4 Fj7 " 30 00 1 0) 40 00.4 4.)) Risch Cjw 110 rx)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~44'~ 44 '-.. 4 tIn;Icr FRirc S~rsueLY (1I 4. L) (virf~unv Secondary)V":P...4. (91 Nper%) ~~~~~~~~~~~~~~~~4-t I0 i1crcceim,:e.- o(flss'willicstic~i In ApplIvrshirsi J t..-74 4 4t- ' +44-l- It 'dI I II414 4' —J b..z y' 7 j.. 4......1r.l. 49,\~j,4.j. "U~~~afearles1)a~~~~~~~4nn2StanL'rJ~~~~~~~i Li-iY#.2.Xp~~~~~~~~lkrnseu~~~~,kkr.4iM. ~~~~~~~4' ''~.44,, -~ ~~~WI.45..4 44 ' P 1 Prifillry scralegy 'IS -Secaondairy Strutegy

I 1 (1) The ripple process (2) The embedding process (3) The bridging process (4) The transfer of technology process (5) The creative application process (6) T!h-' structuring process (7):e stat' S?..-c: I:nod'i.'! t.'iro Ce Figure 1: The seven categories of resarch processes. The shaded areas represent azvailable mowledge based on past research. The unshaded areas represent new knowledge encompassed by the new research.

vigmv,7~: r3t~ig2,g Vr;rm ~ri~k JcT In Gnhiteflheibty nipcim Pulillsitcd hI hiati~ngernnt.'cletce. Qperallons D.'se-trchoi4 n In Iwerroms~ (195241995) I C i.. 4..Jq~4 'hW W6.. Y L m 'loz -. fx&AIkol r Id"Wi, Aptueom sGr(w -1, I Y"W."r,61.I.r;.1 * rz,,- Z Llk- 6 - 'IL - -,I[ -F It - IIA 4 I 1157.1 U a a. I113 ISaI 3 I 14 4~ ~ ~~a I'Ml 1 114 f4 1 1 0 if 4 a 1174 14J 4 I2V 1173 14 4 19711 2I 1 a i 11fe71 JI 1. 127 a 19 14 2 1 1 a ~ I 11.5 I 27 a 0 1 1976 IOU 31 A 2 I 099 151 $i 1 7 a 4 1171 III 33 I 71 U 1 a 2 19544 72 J~I 1 7 a 7 X aav~~~ ii ii 1 4~~1 a 27 1 1, l 13?7- IL I I 1 l I VVI 144. 44 i 5 I S I

FlIjo re J: Trnch of nt11ory an-d A pplicidinn P',perim 411(ime11chiry Pubt4Nhed In 'Managerment SClIMe,te OJr'rafti'ns R-4earch. and lit liderritc-4 (1952- I995) I ~II. I V.ag f Yhc~n ru~~g ALpi4 A2 htl Thke e*la y0 -~~ a I I. g. a.,.I- I I - - I -. 0 2 1 2 a 3 0 0 0 0 0 ~ ~~ ~ ~ ~ ~~.31 IIMI 0 1 1 1.3 ~ 1 a 1 i ~~ ~~ ~~ ~~~~~~3 7 1 aI f 2 3 11 Ile 14 1 1&7 2 4.4 A 13 T4 I 1I4 3 1.1 4 I 4 11 22 ~997.3 a.1.3 11~~~~~~~~~i 1:4 201 25I 107U 1 I I 1....3... 141 21 2 ~~~ *1 1 4 ~~~~~~~~~4 VA 15 2A 30 VIM I1 2 1 24 41 ~~~ 2 1 2 3 ~~~~~~~~~ 2.4 I a 3a1 44 P7at j 2 5 71~ 42.19 17 I3 32 20 44 530 1570 a 1~~~~ ~~~ ~ ~~~~ 32 47.53 1 2. 1 24 51 le 171 2 2 3 4 ~~~ ~~~ ~~~ ~~37 74 5.4 43 15911 ______ 3 9 9 ~~~ ~ ~ ~~~43 9 42 77 1532 3 3 4 11 30 45~~~~~~~~~~~~~~~~A 74 1113 7 2 a ~ ~ 90 O3 22 73 is 11494 1 1 2 J 'Ai 33 75 at 1lls 2. ST.34 77 91 I01 j. 7.1 a too I 3.11 42.14 VA 1001 la 9%V 1 34.1.0 4 511s9 10~~~) 2 3 ~~ 4.4 467 54 t0t ib 1Uf 4 4 7 "l14 1 I I I 0 I I I I I Irij I - - 0 1 - -4 -T i 7 I-L4 -_ ( 71 - I ' 44 1 lei I u - ---