Research Support University of Michigan Business School November 1998 A STATISTICAL COMPARISON OF RESEARCH STRATEGIES PURSUED WITHIN THREE MANAGEMENT SCIENCE SUBDISCIPLINES AND IN MANAGEMENT SCIENCE AS A WHOLE WORKING PAPER #98023 BY ARNOLD REISMAN REISMAN & ASSOCIATES ASHOK KUMAR UNIVERSITY OF MICHIGAN BUSINESS SCHOOL AND JAIDEEP MOTWANI GRAND VALLEY STATE UNIVERSITY

A STATISTICAL COMPARISON OF RESEARCH STRATEGIES PURSUED WITHIN THREE MANAGEMENT SCIENCE SUBDISCIPLINES AND IN MANAGEMENT SCIENCE AS A WHOLE Arnold Reisman Reisman & Associates Shaker Heights, OH 44122 Ashok Kumar' Visiting Scholar of Operations Management University of Michigan Business School 701 Tappan St., Ann Arbor, MI 48109-1234 Phone: (734) 764-4713 Fax: (734) 936-0279 Email: Ashok_Kunmar@ccMail.Bus. Umich.Edu Jaideep Motwani Seidman School of Business Department of Management Grand Valley State University 310 W. Fulton, #718 Grand Rapids, MI 49504 KEYWORDS: Meta Review, Game Theory, Research Strategies, Application Content Rating. November 3, 1998 1 Contact Author. Comments and suggestions on this draft are most welcome.

I

I ABSTRACT In this paper we compare the findings resulting from content analyses of three Management Science (MS) subdisciplines - Flowshop Scheduling/Sequencing, (FSS), Cellular Manufacturing (CM) and Game Theory (GT) - and that of MS in its entirety. In the case of FSS and CM the analysis included the entire life cycle literature. In the case of GT the analysis included all articles published on the subject over the lifetime of the three flagship MS journals. In all, 855 articles, published between 1952 and 1994 were reviewed and classified firstly, on a scale ranging from pure theory to bona fide application. Secondly, the articles were classified in terms of seven types of research processes used by authors. Next, statistical correlations were performed relating data from the above two classifications. The findings show that the literature is dominated by what Ormerod and Kiossis (1996), call Untested Theory. However there is great variability in this regard between the respective subdisciplines and between each and the field as a whole.

I INTRODUCTION In this article, we investigate the trends and directions that the research in Management Science (MS) as a discipline has taken over last four decades or so. Our investigation looks at two separate streams of interest: First, we look at the research strategies employed by the authors in breaking new grounds in the area of MS, and second, we look at the application/theory content of publications in this area. Before proceeding further, we indicate that we use the term MS in generic sense, i.e., it includes all areas of interest to operations researchers, operations managers, industrial engineers, and other quantitative disciplines. Our findings are based on our meta analysis of three sub-areas of Management Science - Cellular Manufacturing, Flowshop Scheduling, and Game Theory, although, we suspect that the findings reflect a more general trend applicable to all MS research. "OR/MS on Thin Ice", Lillien (1970); "OR A Post Mortem", Ackoff(1987); "Where Have we Lost our Way", Reisman (1987). The titles of these articles paint a very bleak picture of and for MS. Their dates of publication speak for themselves. But times were fairly good and higher education in the US was still inan expansionary mode. So, other than providing fodder for some hotel lobby or hallway conversations at various professional society conventions such as INFORMS, CORS, IFORS, IIE, and DSI, these concerns and warnings were quickly forgotten. Nothing was changed. Although at least the first two of the above authors can be recognized as heavyweight contributors to the state-of-the-art of, all of these papers were based on perception and on anecdotal information. In the last five years however, a body of literature has emerged in support of I

I the above assertions yet, based on what some will consider, fairly good MS on MS. Among these are: Corbett and Van Wassenhove (1993); Reisman and Kirschnick (1994, 1995); Reisman et al. (1997 a, b and c); Ormerod and Kiossis (1996); Gupta (1997); Chandrashekar and Kleinsgore (1997). Inasmuch as MS is a management discipline it should be mentioned, if but parenthetically, that a body of similar meta research is emerging for management as a whole as well as for other management disciplines. Thus, Doyle and Arthurs (1995) have applied citation analysis to judge the quality of research in management departments of business schools. Citation analysis has also been applied to judge the quality of journals; for Interfaces, Gupta (1997); in the technology innovation and management area, Cheng et al (1997); in artificial intelligence, Cheng et al (1996); and in the MIS area by Holsapple et al (1993 and 1994). Although a number of peer surveys have been published over the years, for example Liker (1995) and Jones et al. (1996a) their reliability is coming into question, Doyle et al (1996), Cheng et al. (1997). To be sure, citation analysis too is not without its critics as for example Jones et al..(1996b). In his paper Ackoff(1987), referred to the historic trends and the current status of Operations Research (OR) as a "devolution" if not self destruction. Others were and continue to be much more optimistic. Blumstein (1987) for instance, envisaged a bright future if the members of the profession/discipline did more "missionary work". He for one did exactly that over the years in meaningfully applying MS to structuring and solving significant problems within the criminal justice system. Larson (1988) did the same kind of missionary work in the waste disposal field. Yet, with each generation of Ph.Ds in this field and their feedback into the ranks of academics, OR, MS and even OM 2

I are becoming ever more focused Reisman (1987, 1998 and 1992) - a natural trend according to Corbett and Van Wassenhove (1993). As will be shown in this paper in some MS subdisciplines these cadres have been allowed to establish a paradigm and to inflexibly apply it to what is published in the flagship journals. Moreover, this paradigm is reflected in what is being taught in various educational programs. A result of this trend is affecting the availability/offerings of MS subject matter within the core and the elective curricula offerings in US schools of business as documented by Chandrashekar and Kleinsgore (1997). In turn this is reducing the MS literacy of future managers or what Papagieorgiou (1997) calls the OR/MS client base. Thus, the vitality of MS and its relevance is being seriously, and perhaps irrevocably, affected. As evidenced from other disciplines the tendency, for what Corbett and Van Wassenhove (1993), called a "natural drift" to change the openness course will certainly be there, and indeed it is there in MS as well as in Operations Research (Reisman and Kirschnick 1994, 1995, Reisman et al. 1996 a and b). Moreover as shown by Ormerod and Kiossis (1996), this tendency is not strictly a phenomenon that is contained by the shores of North America. Nor is it a recent phenomenon as illustrated by a UK based profession known as Sarjeants at Law. A profession which is now extinct precisely for these reasons (Abbott 1988). For the sake of the continued vitality and relevance of MS worldwide may that not be allowed to happen (Abbott 1988). We must recognize that a counterculture is taking root in MS academic circles worldwide. The upstart International Journal of Operations and Quantitative Management, has been and continues to run a series of Open Forum papers describing alternative approaches to the teaching of MS in business schools. Among these are 3

I Pollack-Johnson (1996), Lam et al (1997) and Rabinowitz et al (1997). Interfaces and OR/MS Today, among others, over the years have also carried a number of papers addressing these issues (Evans 1993, Homer 1995, Powell 1995, and Trippi 1996). At various stages of a discipline's development there is a need for some introspection, based on meta research, to appraise the discipline's status in as objective and all encompassing way as possible. However as was demonstrated by the authors of this paper for the case of the Technology Management literature (Cheng et al. 1997) and shown by Gordon (1982) in a more general context, subjective surveys no matter how professionally executed are sometimes at variance with the facts on the ground. This paper, though not original in the methodologies used, is the first piece of meta research providing a content analysis of the literature encompassing three disparate MS subdisciplines as well as MS as a whole. APPROACH USED IN THIS RESEARCH ON RESEARCH As indicated earlier, this paper derives from five others (Reisman and Kirschnick 1994,1995, Reisman et al. 1997a, 1997b and 1997c) with a common approach to data collection and analysis. In three of these papers we provide a statistical review of the entire life cycle literature, respectively, of FSS, CM and GT using the same categories as those used in the two Reisman and Kirschnick papers cited earlier, both of which look at all papers published in a complete volume-year within Operations Research, Management Science and in Interfaces at two points thirty years apart. Specifically, all articles were first classified according to which of the following research strategies were used by the respective authors. 4

Definitions of Research Strategies' Ripple An extension of previous theoretical or applied type of research in a given discipline or sub discipline. Embedding Bridging The development of a more generalized formulation or a more global theory by embedding several known models or theories. The bridging of known models or of known theories resulting in the growth of the contributing and/or some initially unrelated field of knowledge. Transfer of Technology The use of what is known in one discipline to model problem domains falling in some other, perhaps even in a disparate discipline. Creative Application The application directly, not by analogy, of a known methodology to a problem or research questions not previously addressed. Structuring The process of organization and documentation of the organizational phenomena in the form of models. Statistical Modeling Models which arise from analyses performed on empirical data. These models arise from statistical manipulations such as regression or cluster analysis rather than from logical derivations based on various assumptions. The above strategies are graphically depicted in Figure 1. A Scale for Classifying Theoretical and Applied Articles As indicated in Reisman et al. (1997b), the literature of MS in general, and that of FSS in particular, uses the word "application" to imply anything from a bona fide solution of a real world problem, to an interesting model which is but a figment of the authors' imagination. Moreover the word "data" is often used in referring to numbers 5

I extracted from a random numbers generator. Consequently, the articles comprising the data base for this paper were also classified using a scheme developed in Reisman and Kirshnick (1994). Accordingly, first each paper was judged to be either theory or application. Papers falling into the first group are formal constructs which are theoretical in nature. They may be motivated by or even based on real world problems and offer a wide range of potential applications. Yet, the authors have failed to demonstrate specific examples. Each theory paper was subclassified to distinguish between those which used synthetic numbers for various tests or examples (Theory with Synthetic Data - labeled T2 in the tables) and those that did not (Pure Theory - labeled T1 in the tables). In the application of the above scale synthetic data were defined as outputs of random number generators or numbers created by the researcher for purposes of testing or demonstrating a model. On the other hand, if the paper was judged to fall in the applications area it was then classified on the following five point scale: Al = A figment of the modeler's imagination, a result of logico-deductive reasoning. A2 = A figment of the modeler's imagination that uses synthetic data. A3 = A grounding in the real world. A4 = A grounding in the real world with real world data and a demonstrated application that made a difference. A5 = Either category 3 or 4 above with the additional use of synthetic data to test sensitivity, conduct an error analysis, and/or explore behavior boundaries. Additionally, as in Reisman and Kirschnick (1994) within both the Theory and the Applications subsets we looked for articles which predominantly addressed the following: Meta Research operationally defined as research on, respectively, FSS, CM, GT and MS research in general, moreover, Philosophy/History was operationally defined 6

I as work dealing with the historical and/or philosophic aspects of MS and each of the above MS subdisciplines research. Both of the above sets of classification data were statistically correlated. In the case of FSS, CM, and the GT studies we invoked one modification to the Reisman and Kirschnick papers cited. Articles which use a previously published scheduling model and proceed to improve the solution technique without adding to the model's real world validation were classified as theory whereas in the earlier study it may have been counted as an application depending on the level of real world grounding in the original article. RELIABILITY AND VALIDITY ISSUES' Reliability of Ratings To check on the issue of consistency of the above subjective ratings/classifications in the case of the FSS study we have taken a 10% random sample of the literature e.g. 17 articles and asked for two classifications to be made independently of the authors of that paper. One of the outside classifiers had much experience in this process as he was intimately involved in the classifications leading to Reisman and Kirschnick (1 994 and 1995). The other, albeit a co-author of a related piece of meta research (Reisman et al 1997a), was not personally involved in the classification process prior to doing this task4. The results are as follows: Dr. Cheng's results were identical to ours on 14 out of the 17 articles. In one he differed from us on but one item. Namely, he classified the paper as applications category 4 versus our classification of ' Adapted from Reisman et al. 1997b 7

I category 3. In another paper he judged our secondary strategy to be his primary, all else the same. In the third paper he assigned creative application as the primary strategy while we judged it to involve the transfer of technology. Frank Kirschnick, on the other hand, fully agreed with us in 16 out of the 17 papers in terms of theory versus application. The one difference had to do with whether or not synthetic data were indeed used. In terms of research strategies there was complete (both primary and secondary strategy) agreement in 14 out of the 17 classifications. In one case of the remaining three we concluded that bridging was primary and creative application was secondary whereas he judged creative application to be primary and found no secondary strategy. In the second case we differed only on the secondary strategy. We felt it was bridging. He judged it to be transfer of technology. In the third case our primary classification was creative application and secondary was structuring while his primary was structuring and secondary was ripple. The fact that both the classification schema and the rating scale were applied in two previously published studies by Reisman and Kirschnick (1994, 1995) on a wide OR/MS literature base, and in another subdiscipline in-depth study by Reisman et al (1996), should not be overlooked as a further quality control check. In addition, two Britishers "adopting the same method..."(Reisman and Kirschnick 1994), wrote a paper which "looks at the same issues through the analysis of UK based journals" as well as the European Journal of Operational Research (EJOR). Apropos, Ormerod and Kiossis, (1996) say the following: "To check that we understood the method and its application by the originators we conducted a pilot study. For the pilot study we analyzed Interfaces in 1972 and 1992 following the definitions and conventions of the originators of the 8

I method. Comparing the results of our analysis with those published we obtained very similar results." They then proceed to document the above assertion. Thus we have addressed the subjective rating/ classification consistency issue. However, a question which arises almost instinctively is that of validity of the research instrument used e.g., both the scale and the classification schema. This issue will be addressed in discussing each element of the research instrument going from the most transparent element to the most opaque. The issue of "data" is the most transparent. Authors either presented numbers or not. If they did we looked for the data's source. If there was no indication of obtaining these numbers directly or indirectly from some real world organization we concluded that these "data" were synthetic. In this regard there was rarely any question among the evaluators. While in this process we also looked for any real world grounding/motivation specific to the research being reported on. If none were to be found we concluded that it was theory irrespective of any application claims by the author(s). Work with or in Company X or institution/organization Y was judged as an application. Here, however, we were sensitive to any disguises that might have been invoked, hence in a few instances it was a judgment call. Philosophy/History articles by their nature were easy to pinpoint. In classifying the research strategy used the ripple process was the easiest to recognize. In this respect the authors themselves were very helpful. There was much prior literature to be cited and it all was directly relevant to the problem being addressed. On the other hand, the structuring process was easy to identify for opposite reasons. There was a paucity 6f directly relevant analytical or MS type literature to be cited. Instead the authors typically cited descriptive, institutional or contextual literature. 9

I The creative application process was also relatively easy to spot as the author clearly applied a well-established MS methodology to a problem not previously so addressed. In the use of the embedding process the authors were helpful in telling us that they have created a model etc., that is more general then its constituent parts obtained from previously published literature. When transfer of technology was invoked the authors were helpful. They often said as much and the publications they cited were from other, often disparate, disciplines or model domains. The bridging process was the least transparent to identify as it was sometimes, on first reading, confused with embedding or the transfer of technology. Although the statistical modeling process is easy to identify, as will be shown later, in the FSS and the CM contexts the point is mute. It should be mentioned that at times the authors used more than one of the above strategies. In these cases we have identified the primary and the secondary such strategy. Validity of Ratings The issue of model/theory validation has been of concern for many generations in the general literature of science. This concern has not escaped the literature of MS. The following two sections are abstracted from Reisman et al (1996). In 1993 the EJOR dedicated a special issue to this subject. Specifically, Little (1970), Roberts (1977), Eilon (1979), Powers et al. (1983), and Gratwick (1983), among others, argue that validation should be based on the criteria of simplicity, transparency andflexibility. Landry et al.(1983), and Banville (1990) argue that a valid model is a legitimate model. While, Toulmin et al. (1979) and McCloskey (1985), argue that in MS a convincing argument may serve for model validation. And there are always "the correspondence criteria that measure the degree of conformity of the model to empirical 10

1 facts" "put forward by such OR pioneers as Rowe, Williams, Blackett, Waddington, Morse, Kimball and Koopman." (Dery et al., 1993). In the Reliability of Ratings section we have shown that our methodology is simple and transparent to the point in some aspects that it is intuitively obvious. It is flexible in that it has been applied to other, rather different literatures, and by other investigators. Its legitimacy is addressed by several of what we consider to be convincing arguments. Lastly it corresponds to the views, based on a wealth of empirical observation and reflection by many very seasoned contributors to MS theory, practice and education e.g., Lillien, (1985), Ackoff(1987), Blumstein (1987), Miser (1987), Geoffrion (1992); to the findings of other contributors to meta research on MS, Corbett and Van Wassenhove (1993), as well as those of an impartial scholar of professions in general, (Abbott, 1988).5 DISCUSSION Because the numbers differed across the various studies in this paper we basically show results in terms of percentages. Table 1, provides the comparative statistics for each of the subdisciplines studied and for the 1992 volume year of the flagship, US based, MS journals. It also provides the combined data. Figure 2 provides an alternative display of some of the same statistics. These results represent analysis and codification of no fewer than 855 articles. As can be seen FSS researchers invoked the ripple strategy the most. 71.8% of the papers were judged to use this as a primary strategy another 6.5% used it as a secondary 11

I strategy. GT authors publishing in the three above mentioned journals invoked this strategy the least although 47.9% in the primary mode and 8.3% in the secondary mode are non trivial levels. CM authors at a 56.6% primary use level and 11.1% secondary use level were incomparable from MS authors at large, e.g., 56.5% and 8.8% respectively. GT represents the largest percentage (38.2% in primary mode and 5.6% as a secondary strategy) of papers invoking creative application (CA) as a strategy. This is well beyond the corresponding figures of 3.3%/10.5% across all MS subdisciplines for the 1992 volume-year of the three journals identified earlier. It is also more than three times the usage incidence found in the entire literature of FSS and more than twice that of CM. The reason for this may well be that GT is viewed in MS more like a methodology e.g. Linear Programming than as problem area as is the case of CM and FSS. However, as will be shown later not all papers using CA as a research strategylall into the true (real world) application category. In fact, the structuring strategy was almost non existent (0.7%) among the authors comprising the GT literature subset reviewed. This fact seems paradoxical in light of GT being considered as a methodology. Although structuring is the essence of the kind of "missionary work" that Blumstein (1987) called for as a primary strategy, it has a higher incidence of use in FSS (6.5%/0.6%), and CM (6.4%/5.1%), than it does in MS as a whole, where it was primarily invoked only in 2.9% of the papers. The embedding, bridging, transfer of technology, and statistical modeling strategies combined accounted for no more than 8.9% of the FSS literature; 13.2% of GT; and 20.8 of CM. Moreover, CM recorded no statistical modeling papers, in the primary mode FSS only 0.9%, and GT 2.9% as compared to 2.9% for MS as a whole. 12

I As discussed in Reisman et al (1996), and indicated in Table 2, and in Figure 3 over its life time, now in its fifth decade, the entire literature of FSS has recorded only four articles (2.4% of total) that were judged to be true or tested applications e.g., applications categories 3, 4 and 5 combined. This rate is significantly below the 30% (based on page count) rate recorded for the three flagship US based MS journals at two points (three decades apart) in time (Reisman and Kirschnick 1994). It is also significantly below the 14% rate in 1994 for the major UK based journals (Ormerod and Kiossis 1996) and it does not compare well, in this regard, to other MS subdisciplines e.g. 15.3% in Cellular Manufacturing6 (Reisman et al. 1997a), and 20.8% in Game Theory7 (Reisman et al 1997b). Table 2 also shows that CM recorded the largest percentage (56.6%) of papers addressing theory with synthetic numbers with FSS following closely with 51.8%. In contrast GT had a significantly lower percentage (17.4%) of papers in this category. On the other hand, GT had the distinction of recording the largest percentage (31.3%) of pure theory papers and the largest percentage of untested e.g., applications categories I and 2 papers. This is interesting since GT, as shown earlier, also recorded the largest percentage of true applications. Table 3 provides the statistics to support a point made earlier in the paper namely, that the use of creative application (CA) as a reresearch strategy does not imply a solution of a real world problem. In fact the columns of Table 3 represent each of the designations ranging from pure theory to a category 5 application. The rows represent the three subdisciplines studied. Thus, it can be seen that FSS has no record of CA being used in any of the application categories (3,4 and 5) that involve real world data. This is not surprising. The point is better made by the subset of FSS literature representing a 13

I combination of CA use in a theoretical construct that uses nothing but synthetic data. This subset accounts for 50% of all FSS papers that invoke CA. Lastly, Ormerod and Kiossis, (1996) extended the Reisman and Kirschnick (1994) content analysis to flagship MS journals which are based in the UK and also to the European Journal of Operations Research. However they went a step beyond and "included the country of affiliation of the authors" so as to provide "data on the publishing habits of United States affiliated authors". Because the FSS, CM and GT literature is so rich in untested theory (categories T1, T2, Al and A2) we examined the geographical distribution of its contributors and its host journals. Our findings for FSS show that 33 of the 170 articles (19.4%) appeared in the very journals examined by Ormerod and Kiossis. Of these, however, 20 or 60% were written by US affiliated authors. Of the remaining 137 articles, 109 or 79.6 % were written by US affiliated authors. In the aggregate these authors contributed 129/170 or 75.9% of the FSS literature. Thus at least the FSS literature supports Ormerod and Kiossis' finding of "the increasing role played by untested theory in publications by United States affiliated authors" especially so by those who have sought out and succeeded in having their work published in the European journals. CONCLUSION "Periodically professionals in a given field must step back and assess where the field has been, where it is heading, and what if any, should the scientific community do to change the course" (Gupta 1996). The main objectives of this study were to investigate the MS subdisciplines of FSS, CM and GT over the entire lifespan for the first two and 14

I over the lifespan of the flagship US based MS journals in the case of GT; to obtain a succinct, quantitative, yet operationally meaningful summary.. Comparison of the results obtained for FSS with CM, GT and with MS in general, leads to the conclusion that FSS research is in dire need of a paradigm shift to enhance its probability of ever becoming a tool for the practice of MS. This shift may include enlargement of the field's scope as well as diversification of research strategies pursued including immersion of some researchers in problems of the real world (Reisman et al. 1997b). Albeit less emphatically the same can be said for research in CM (Reisman et al (1997a). With all due respect to choice recently made by the Nobel Prize committee our analysis of the GT literature recorded in the lifetime of the three US based flagship MS journals leads to the conclusion that GT is a basically a stimulus to a way of thinking. "It is not yet an item in the decision maker's tool box and its advances in the literature surveyed tend to be rather marginal and theoretical". Reisman et al (1997c). The MS literature in general should do much more to seek out and recognize work that is satisfied with perhaps less elegant but implementable, better yet implemented solutions. Lastly, "as can be seen, the model validation issue cannot be taken for granted. It even is a topic of debates in OR." (Dery et al. 1993). The authors of this paper will be the first to admit that we may not have taken all possible measures to validate both our classification scheme and/or our data collection. We have however done more in this regard then the bulk of the extant MS literature in general and that of Flowshop Scheduling /Sequencing, Cell Manufacturing and Game Theory in particular. 15

I REFERENCES A. Abbott, The System of Professions: An Essay on the Division of Expert Labor. University of Chicago Press, Chicago, pp. 118-121, 1988. R. Ackoff, "OR: A Post Mortem", Operations Research, 35(3), pp. 471-474, 1987. C. Banville, "Legitimacy and Cognitive Mapping: The study of Social Dimension of. Organizational Information Systems" Ph.D. Dissertation, Laval University, Quebec City, Canada. 1990. A. Blumstein, "The Current Missionary Role of OR/MS". Operations Research, Vol. 35, pp.926-929. 1987. C.H. Cheng, C.W. Holsapple, and A. Lee, "Citation Based Rankings for Artificial Intelligence Research: A Business Perspective", AI Magazine, 17(2), pp, 87-96, 1996. C.H. Cheng, A. Kumar, J. Motwani, A. Reisman, and M. Madan, "A Citation Analysis of the Technology Innovation Management Journals" Under Second Review in IEEE Transactions on Engineering Management, 1997. C.J. Corbett, and L.N. Van Wassenhove, "The Natural Drift: What Happened to Operations Research?", Operations Research, Vol. 41, pp. 625-640, 1993. R. Dery, M. Landry, and C. Banville, "Revisiting the Issue of Model Validation in OR: An Epistemologic View," European Journal of Operational Research, Vol. 66, (2), pp. 168-183, 1993 J.R. Doyle, and A.J Arthurs, "Judging the Quality of Research in Business Schools: The UK as a Case Study", Omega, Vol.23, pp. 257-270, 1995. J.R. Doyle, and A.J Arthurs, L McAuley, and P.G. Osborne, "Citation as Effortful Voting: A Reply to Jones, Brinn, and Pendlebury, Omega, Vol. 24, pp 603-606, 1996. S. Eilon, Aspects of Management, Pergamon Press, Oxford, UK, 1979. J. Evans, "Creativity in MS/OR: The Multiple Dimensions of Creativity," Interfaces, Vol. 32(2), pp. 80-83, 1993. A.M. Geoffrion, "Forces, Trends, and Opportunities in MS/OR." Operations Research. Vol. 40, pp.423-445. 1992. 16

I U.G. Gupta, "Using Citation Analysis To Explore The Intellectual Base, Knowledge Dissemination and Research Impact of Interfaces (1970-1992)", Forthcoming in Interfaces, 1997. C.W. Holsapple, L.E. Johnson, H. Manakyan, and J. Tanner, "A Citation Analysis of Business Computing Research Journals", Information and Management, Vol. 25, pp. 231-244, 1993. C.W. Holsapple, L.E. Johnson, H. Manakyan, and J. Tanner, "Business Computing Research Journals: A Normalized Citation Analysis", Journal of Management Information Systems, Vol. 11(1) pp. 131-140, 1994. P.R. Homer, "Practice Makes Perfect: Two Courses that Bridge the Gap," OR/MS Today, April, p. 24, 1995. M.J. Jones, T. Brinn, and M. Pendlebury, "Judging The Quality of Research in Business Schools: A Comment from Accounting", Omega, Vol. 24, pp. 597-602, 1996a. M.J. Jones, T. Brinn, and M. Pendlebury, "Journal Evaluation Methodologies: A Balanced Response" Omega, Vol.24, pp. 607-612, 1996b. B. Pollack-Johnson, "Connecting Quantitative Methods Education to the Real World: Teaching Formulation, Modeling, and the Process of Problem Solving". International Journal of Operations and Quantitative Management, Vol. 2(1), pp. 73-81, 1996. R.L. Keeney, "Using Values in Operations Research". Operations Research. Vol. 42(5), pp. 793-813, 1994. K. Lam, K.K. Lai, and J. Wiest "Meeting the Neeeds for OR/MS in Hong Kong", International Journal of Operations and Quantitative Management, Vol. 1(3),23 5-241, 1995. M. Landry, J.L. Malouin, and M. Oral, "Model Validation in OR", European Journal of Operational Research, Vol. 14, pp. 207-220. R.C. Larson, "Operations Research in the Service Industries". In Managing Innovation: Cases From Service Industries, B.R. Giles and J.B. Quinn (eds.). National Academy Press, Wash D.C., pp. 115-143, 1988. J.C.Little, "Models and Managers: The Concept of a Decision Calculus", Management Science, B 16, pp. 466-485, 1970. G. Lillien, "OR/MS On Thin Ice", Interfaces, Vol. 15, (4), pp.12-13, 1985. 17

I J. Liker, "Results of a Survey of Management Journals for TIM Research" TIM Newsletter, Academy of Management, Vol. 7(2). pp. 5-8, Winter 1995. D.N. McCloskey, The Rhetoric of Economics, The University of Wisconsin Press, Madison, WI., 1985. H.J. Miser, "Science and Professionalism in Operations Research", Operations Research, Vol. 35, pp.314-319, 1987. R. Ormerod and I. Kiossis "OR/MS Publications: Extension of the Analysis of the US Flagship Journals to the UK. Forthcoming in Operations Research. 1996. J. Papageorgiou, "Are We Promoting OR/MS to Our Future Clients?" Forthcoming in International Journal of Operations and Quantitative Management, 1997. S. Powell, "The Teacher's Forum: Six Key Modeling Heuristics," Interfaces, Vol: 25(4), pp. 114-125, 1995. R.F. Powers, J.J. Karrenbaner and G. Doolitle, "The Myth of the Simple Model," Interfaces, Vol. 13, pp. 89-91, 1983. G. Rabinowitz, Y. Edan, and A Mehrez, "CIM Laboratory Decision and Implementation Process", forthcoming in: ". International Journal of Operations and Quantitative Management, 1997. A. Reisman, "Where Have We Lost Our Way?" OR/MS Today, 1987, October, p 7. A. Reisman, "On Alternative Strategies for Doing Research in the Management and Social Sciences". IEEE Transactions on Engineering Management. Vol. 35(4), pp. 215-220, 1988. A. Reisman, Management Science Knowledge: Its Creation, Generalization and Consolidation. Quorum Books, Westport, CT., 1992. A. Reisman, 'OR/MS Education: A Need for Course Correction," International Journal of Operations and Quantitative Management, Vol. 1(1), pp. 67-76, 1995. A. Reisman, A. and F. Kirschnick. "The Devolution of OR/MS: Implications From a Statistical Content Analysis of Papers in Flagship Journals". Operations Research. Vol.42(4), pp. 577-588. 1994. A. Reisman, A. and F. Kirschnick. "Research Strategies Used by OR/MS Workers as Shown by an Analysis of Papers in Flagship Journals". Operations Research, Vol. 43(5), pp. 731-740. 1995. 18

I A. Reisman, A. Kumar, J. Motwani and C.H. Cheng, "Cellular Manufacturing: A Statistical Review of The Literature (1965 - 1995)", Operations Research, Vol. 45, No. 4, p. 508-520, 1997a. A. Reisman, A. Kumar and J. Motwani, "Flowshop Scheduling/Sequencing Research: A Statistical Review of the Literature 1952-1994." IEEE Transactions on Engineering Management, Vol. 44, No. 3, p 316-329, August, 1997b. A. Reisman, A. Kumar and J. Motwani, "Game Theory Literature: A Meta Review Qf Its Application Claims", Working Paper. 1997c. E.B. Roberts, "Strategies for Effective Implementation of Complex Corporate Models," Interfaces, Vol. 8, pp. 26-33,1977. S.T. Toulmin, R. Rieke, and A. Janik, An Introduction to Reasoning, Macmillan, New York, 1979. R. Trippi, "The AIM Game: Learning Investment Management Principles through Monte Carlo Simulation," Interfaces, Vol. 23(3), pp. 66-76, 1996. FOOTNOTES 1. Sources: Reisman (1988a) and Reisman & Kirschnick (1995) 2. ibid. 3. Frank Kirschnick is now a Doctoral Candidate at the Engineering Economic Center, Stanford University. 4. Chun Hung Cheng is a Lecturer in the Department of Systems Engineering and Management at the Chinese University of Hong Kong. 5. We are grateful to one of the anonymous reviewers and especially to the Department Editor of the FSS paper for encouraging us to address the issues of reliability and validity. 6. 36 papers in Applications categories 3, 4 and 5 out of a life cycle total of 235. 7. This percentage reflects only the Game Theory articles that were ever published in Operations Research, Management Science or in Interfaces. 19

Table 1: Compara tive Analysis of Papers Published in Flow-Shop Scheduling, Cellular Manufacturing, GCame Theory, and Flagship OR/TS Journals Baisled on Research Strnategies Used 'o<!l N I.1% Ripple i ltuhleddimi it jlritging.i r W illechi. ICre.ttive AppI'r..St(ncluring IStlt. tideling qp I k or 'anpers, s ~ I I s I I-, I s - I -) s I I., i -- I $ r PJ S lotal No. 1of Papers (P or ) 170 122 1 11 6 2 3 7 4 2 22 2( 11 I 2 1 Flow Shop Schedluling '/o of 'apers (P or ) 10% 71.8 6.5 13.5 1.2 1.8 4.1 2.4 1.2 12.9 11.8 6.5 (0.6 1.2 j 1952-94 No..of I'.&SCiassirications 213 133 8 11 6 42 12 2 % 6of' & S Clftssfic(lons (100/a 62.4 3.8 4.7 2. 19.7 5.6 1.9 otal No. or Papers 1' or S) 235 133 26 9 27 1 6 13 7 38 17 15 1 (1 Cellular M u factur'ing / t'apers P r S) 56.6 11.1 7 11.5 I 2..53.08 16.2 7.2 6.4 t 5. I (1. 195-95 No. ori'& S Clssifirations 307159 13 33 201 55 27 (1 % Aofa & S C:!ussicif - ti q 51.8 4.2 1(1.7 6.5 17.9 8.8 (j.(~ T'otd lNo. ofPiIapen (P or S) 144 69 3 3 1 1 8 2 55 12 3 2 4 J 5 C:ame ThecCrv % apes ( or S) 100 47.9 8.3 2.1 2.1 7.6 5.6 0.7 1.4 38.2 5.6 0.7 1.4 2.8 2 1952-95 No.ofal'&S Clhsiications 182 8I 6 19 3 63 3 7 % 4ofP& &.S Clssifticat~(om 100%/. 44.5 3.3 10.4 1.6 34.6 ".6 3.8 OR/MS Flagship i 'l'om NO. of Papers (1' or S) 306 173 27 1 1 3 1 8 7 3 1() 32 9! I Journals SL;ti.ftics % ofl' Papers (P r.S) (100%/ 56.5. 472 4.2 4.2 2. - 25. 9.8 3.3 1 2.9 (10.3 2.9 2.9 1992 No. [MA, S (Clsslilcttios3 426 200 26 21 109 42 11 18 o "f1' of &.S Classifircations m1/ 46.9 6. 4.9 25.6 9.9 2.3.2 Total No. of Papers (1' or S) 855 497 r 76 31 22 54 29 937 41 125 77 36 16 15 12 Cumtlative Statistics % uiIr''Purs ( orS) 100%/a.1. 3.6 2.6 6.3 3.4 11.3 4. 14.6 9. 4.2 1.9 1.8 1.4 No. of P & S Classifications 1128 573 53 83 138 202 52 27. o:. ' &. S CaI S flctl., 100% 50.8 4.7 7.4 12.2 7.9 4.62.4 I _I I t P* = Primary Strategy S = Secondary Strategy

Trdile 2: Comtparative Aall'Lysls orrlwory/Appyicatiun Papers Pulblished in Flow-Shsop Scheduling, Cellular Manufacturing, anmd Game Theory.:_Applications __ __ __ r Theory.- _ __ _ _ __ _ I_ TOM I Cattizory Mtt2 Research lidwsDphyllitsfory Psirf, M) svikthetic bala rll) Mt-ta lzr-%r!llrll I PhU(nonlivil Illuirv I __________ d~~~~ofPapen Al 1A21A31A41 AS 1P IS P) S ' S T Iof otupin (P1Ur S) 170 4j 30 2 j1 ~ 1. 0 1 0 1 34 1 88 I 6 10 1 Flow Sktup schedluling l. t 5spr (1 or S) 100% 2.4 17.6 j LI 0.6 j 0.6 0.0 j 0.60. 20.0 0.6 51.8 ~ 3.5 5.' ) 0. 1952-94 No. ors',& a Ciussierbcl os 191 3H 1 1 35 94 20 2 %otP 4S;Ctassftksokjn; 100% 19.9 0.5 0.5 18K.3 49.2 1 0.5 1.0 lsdu ItKu.utfP0luien (or S) 235I 2 6 5 7 5 J 2 5 1 29 5 133 J 16 7 Cellular Milufiriudluig V. hi taperi (P'or-S) 100%1/ 9~4f7:T 2-6TJTF7T.r 2.1 J 0.9 2 1 0.4 12.3 ( 2.1 5 6.6 J 0.0 6.v. - 1965.95 No. uf la S Ctmusstckwtlaos 255 44 7 6 34 133 23 V4uOFrt.4SC stluly~ 100% 1143 3,0 - 2.6 14.5 56.6 9.8 3.4 Tofsid No, (1'pers(I'orS) 144 24 14 it 10 9 01 0 0 jO 4 0 25 0 4~ 0 21 Caine Theory i aofra pen (ror1.s) 100% 16.7 9.7 7.6 6.9 6.3 0.0 j 0.0 0.0 0.0 313. 17.4 J 0.0 2.nj 0.0 1.4 1952.95 N.of 1'& S Cluotiftrattouis 149 68 0 0 45 25 4 7 ti of I' & S ChNIttcslous 100% 45.6 0.0 0.0 30.2 16.8 2.7 4.7 Ttulltico. or h prsm (P1or$) 549 501 48 1 9 16 17 5 3 5 2 108 6 246 6 30117 511 Cunwlublvet M11Aol~cs 4 VoPupers(PeeP).., 100%.1.7j 35.0 0.9 j 0.5 0.9 0.4 19.7 L, 44.11 1.1 3.3jI 0. N~o.of p &s Ctpuvteutirss 595 150 8 7 114 252 47 17 of P -'S;CtussW1utk1oos~: 100%1/ 25.2 1.3 1.2 124.4792.9 )r I I I 5 II I r- - rnmuwy auuvac~ a - Secondary SunoegLy

Table 3: Distribution of Creative Application Strategy Across the Theory/Application Classifications CA & AI CA & A2 CA & A3 CA&A4 CA&AS5 CA & TI CA & T2 TOTAL FSS Number 1 7 3 11 '22 Percentage 4.55 31.82 13.64 50.00 100.00 CM Number 1 1 3 4 28 38 Percentage 2.63 2.63 2.63 7.89 10.53 73.68 100.00 GT Number 16 9 5 4 5 8 8 55 Percentage 29.09 16.36 9.09 7.27 9.09 14.55 14.55 100.00 a ~~~~~~~ ~ a a a. al a a a a...

I (1) The ripple process (2) The embedding process (3) The bridging process (4) The transfer of technology process () The creative application process (6) The structuring process (7) Thle statistical modeling process Figure I: The seven categories of research processes. The shaded areas represent available knowledge based on past research. The unshaded areas represent new knowledge encompassed by the new research. Source: Reisman and Kirschnick (1995)

Figure 2: Comparative Analysis of the Percentage of Papers Published in Flow-Shop Scheduling, Cellular Manufacturing, and Game Theory Based on Research Strategies Used lipl)le En cddtling Britdging Tnrns. ol rcch.l Creative AIpplI. Structuring I Stat. M~od. _ ___r. or *l. Creative. Struturn __ Sta. M o'.... - 20.0|1,' t ^1-^(^^B'1^::^^^P^ Q^_^?/ Cumulalive Statistics | 0.0-,7X= ffi6S W D^<<5:^ Cellular Manufacturing,Flow S!hop Scheduling 71.8 3.5.8 2.4 12.9 Schedu Cellular Manufacturing 56,6 3.8 11.5 5'5 16.2 6.4 0.0 Gane Theory 47.9 2.1 7.6 0.7 38.2 0-7 2.8 2 j ^ I _ i C1l1ll!:ti ve Statistics 59.0 3.3 7.5 3.3 20.9 4.9 1.1 ~80. __20 CumulativeStati.s 2ics ~~: ii........ ii.....~

Figure 3: Comparative Analysis of the Percentage of Application/Theory Papers Published in Flow-Shop Scheduling, Cellular Manufacturing, and Game Thcoiy 4 $,I_ __ __ &I. V I I -VI. - - V r-v. I 'llcory with Synthletic DaItaI Plure 'l'hcorv Untested Annlications 'l'csted Amnfiic;tilons Others.1 - --.1 - - - -, - - - - -- - - - - - -1.;.......,,...., X* - [ - -.............. Hlow Shop Schlleduling 51.8 20.0 20.0 2.4 5.8 CelluIlar. Mlilt'iactflrinll 56.6 12.3 11.1 7.7 12.3 Game Th'lcolty 17.4 31.3 26.4 20.8 4.1 Cumulative Statistics 44.8 19.7 17.8 9.5 1 8.2 -- ' — '. - _ _ '.... I....' ':-': |, Cum -. ulative Statistics |-10.0 7 g g/ Game Theory,2 g/ Flow Shop Scheduling I,-.:~i~i-i.!:5!:!:., "............. F X H~~ ~... ~~:it;~'. ~ ~~~:: ''e M a n u f a cturing 30. -i; ~. ~-D i