Division of Research Graduate School of Business Administration The University of Michigan NEW PRODUCT EVALUATION: SYSTEM, NOT AN ACT Working Paper No. 427 C. Merle Crawford FOR DISCUSSION PURPOSES ONLY None of this material is to be quoted or reproduced without the expressed permission of the Division of Research. May 1985

I

NEW PRODUCT EVALUATION: SYSTEM, NOT AN ACT New products managers have tough jobs - maybe impossible ones. Their assignment is to: * Bring out a steady stream of big winners, * On time, ahead of time, but never behind time, * With no big failures along the way.. in fact, preferrably with no failures. * Meanwhile, deal effectively with hundreds df interface problems so as to keep new products teams together. * Not be overwhelmed by the flow of new techniques, from tradeoff analysis to simulated test markets to launch control. * All the while increasing productivity, which translated means: cut costs. As if the "impossible dream" of top managements isn't enough, we compound the problem by perpetuating several myths about new product evaluation that make the job even tougher than it should be. Evaluation proceeds simultaneously with creation -- it is a twin streams affair. Creation takes place both on the technical side (the product) and on the marketing side (its marketing plan). So does evaluation. This article speaks to the evaluation portion of this operation. We will look at 'four myths that needlessly exacerbate the evaluation task, which will be followed by examination of several key evaluation concepts which explode the myths and lead to more realistic evaluation processes. Without the myths, new products I

managers have a much better chance of meeting management's impossible demands. MYTH: EVALUATION BEGINS WHEN THE IDEA FIRST APPEARS AND CONCLUDES WHEN THE PRODUCT IS MARKETED. In fact., it begins well before ideation, and goes on for months or even years after marketing. MYTH: A SHARPLY DECLINING DECAY CURVE IS GOOD, THAT IS, WE SHOULD SCREEN EARLY AND TOUGH.. ELIMINATING BAD IDEAS BEFORE WE WASTE A LOT OF MONEY ON THEM. Sometimes yes, usually no. Besides, the thing eliminated probably isn't the idea at all, just an attempt at conceptualizing it. MYTH: PRODUCT INNOVATION IS LIKE A RELAY RACE -- FIRST WE DEVELOP THE PRODUCT ITSELF, AND THEN WE HAND IT OFF TO MARKETING. THAT WAY WE AVOID COSTLY AND TIME-CONSUMING BACKITRACKING INTO THE R&D PROCESS. Less often seen today, but still probably our greatest single deterrent to efficient and effective product innovation. MYTH: THE ONE REAL EVALUATION, AND THE ONE WHICH SHOULD BE DONE THOROUGHLY AND WELL, IS THE ONE WE DO FOR THE GO/NO-GO DECISION. Sometimes a big manufacturing capability or other investment makes this so, but managers often cannot even identify one time when a no-return decision was made. Several key ideas and concepts that lie deep within the product innovation process make these widely held beliefs myths. Let's examine these concepts, to see how they change the above thinking, and what thinking they substitute for it. After that, we will look at some specific implications for managers as they create and implement their new products processes. 2l

THE EVALUATION PROCESS IS CONTINUOUS Odd as it may sound, management is never out of the evaluation mode on new products, for several reasons. First, some of the most critical and decisive evaluation of new product concepts occurs before the concepts are even discovered. More and more managements today are adopting firm and clear new products strategy (product innovation charters) to guide their new products organization. Strategy narrows company focus and thereby eliminates most new product options. For example, when Brik Pak said it would concentrate its asceptic packaging technology only on liquids, that eliminated yogurts, soups, and puddings. Brik Pak's action was evaluation -- a deliberate decision that any new concepts in the rejected areas would not be good ones for that firm. Strategy action such as this is wholesale evaluation, and often very effective. The second reason why evaluation must be continuous is that the new product concept itself is changing. A new product is not born at time of ideation. An idea is, but the manner of its implementation is not. Personally, I hold that a new product comes into existence when it is successfully established in the market place. Before that, a product is actually only a concept in temporary physical form, and even less than physical form if it is a service. This view helps explain the increasingly popular practice of gradually edging a product into the market place. The firm is still exploring -- management is not at all sure the current product is the one they ultimately want. Full-scale marketing, which too many people feel is the ultimate act of product innovation, is actually only one step in the evaluation process. Feedback and tracking

systems should be built in accordingly. A good way for- a management to test this thesis of gradual product evolution is to take a product the firm marketed about a year ago. and reconstruct a list of the many forms it took from the time of original ideation to now. Some firms, e.g. computer companies and copier companies, actually formalize the process, giving names to each step -- breadboard, pe-production model, etc. And they don't" call the current model FINAL, regardless of how close to marketing they are. The third idea which buttrseses the concept that evaluation is a continuous process is that of the will-of-the-wisp GO/NO-GO decision. Let' s don't overstate this matter., because there are projects where management must authorize, say, $100 million for a new plant. That is a clear GO point, and abort after that is extremely unlikely. Too, consumer products companies which launch nationally via television advertising probably consider their TV commitment as a 60 deci si on. But, most firms try to avoid these tramautic points. For example, a key argument for the complex simulation test market models increasingly being used today is that they permit a firm to move into market-by-market roll-outs, rather than a full-scale test market followed by national launch. The industrial firm is trying to get out of such decisions too, one newly popular technique being downstream and upstream coupling. By "renting" the resources of suppliers or potential users, a firm can often edge into a new product venture enough to be sure it will work before making the capital commitment which then breaks the umbilical cords of the coupling. 4

For another reason for continuous evaluation, look at how many firms today view their new product innovation as new business innovation. Either the new product is just one more in a long stream of items in a given business (e.g. Sugar-Free Jello) or it takes the firm into a new business (Jello Pudding). In the former, General Foods has been evaluating gelatin products -for over forty years. In the latter:, the initial product was rightly viewed as the first in a coming stream of pudding and associated products. In fact, the word today is that any really new product should, at the time of launch, have two or three follow-on products moving down the chute right behind it. A PRODUCT AND ITS MARKETING PLAN SHOULD EVOLVE SIMULTANEOUSLY People who support the myth that product and marketing plan play a relay hand-off act are surprised by the statement that in the well managed firm today several of the most important marketing plan decisions are made before the product concept even appears. In that and other ways, marketing comes well before technical research. This is not arbitrary or a mistake. Good product innovation strategy covers more than the area of general focus (mentioned above). It also states the degree of innovativeness sought, and it adds any other parameters important to management at that time. For example, when Pillsbury introduces a new food product next month, we can predict accurately that its marketing plan will have, among other things: * Television advertising. with print support in women/home magazines. * Packaging that carries an important load in shelf identifi r.,

cation and promotion. * Distribution through supermarkets and other -food outlets. t Promotion of the product on its merits, not on price. These features are not surprising, to us or to the product innovation teams at Pillsbury. Procter t& Gamble actually spells them out in printed form, so there will be no mistake. Marketing strategy, by coming first, actually helps drive technical innovation. In the above cases, tightly. But in firms which want fewer fetters on the technical function, marketing strategy constraints are carefully eliminated. The point is, one must give consideration to which is best, for that product team, in that firm, at that time. Whatever the decision, marketing policies should be consciously considered first. And, after that, they are not put away until after technical development is concluded. The hottest organizational device in new products work today is the team -- some variation on the entrepreneurial theme. Not just the traditional matrix team, though that is sometimes the form selected, but more popularly the independent team, or venture team, or just task force. However named, they are separated to various degrees from the on-going business for two key reasons: (1) so they are free of unnecessary institutional constraints in the firm, and (2) so they can be sure that any particular department's task is NOT free of influence by the other departments on the team. This frees the team from the firm, but knits the functions tightly together. One key payoff is continuous evaluation -- making sure that a seemingly simple decision to shift from one gear arrangement to 6

another during technical development doesn't come as a surprise to the technical service group who will repair it and the sales force who will sell it. In reverse, a marketing decision to shift target markets slightly is shared with technical people, who in effect are then helping evaluate the marketing plan. Everyone evaluates everything, continuously, just as they would in a small firm's entrepreneurial setting. EACH EVALUATION DECISION IS IEYED ONLY THE DECIISION/ACTION WHICH IMMEDIATELY FOLLOWS IT. Because new products managers get many shots at evaluating a new item, not just one big one, they tailor their evaluation activity to each of those many points. What is good evaluation activity at one stage is quite wrong at another. Thus, no sacred cows, no "musts'. Just as no particular marketing ploy is good, in the abstract, ditto for test markets, placement tests, conjoint measurement, and the like. The firm, -for- example, which insists on a fullscale screening step prior to R&D either is developing extremely similar new products, or is simply wrong. What determines the proper evaluation activity is the nature of the decisions being made and the immediately subsequent activities they permit. That is, what will happen between now and the next time we stop for an evaluation. Let's look at some examples: Decision: Whether to license coating technology from Du Pont as a starting point in the development. Evaluation:.Thorough concept testing. keyed to the product attributes which will be determined by that technology..7 1

Decision: Whether to begin production of a chewable vitamin tablet on the current vitamin line. Evaluation: Simple product use tests, with product from the pilot process. Decision: Whether to separate the keyboard from the processing unit in a personal computer. Evalaution: Extensive product use testing. The reasons for these are probably quite clear. In the first place, the coating technology will be at the base of the entire development process, so whatever it will do to the ultimate product needs to be checked out. It can be. Second, the vitamin product will entail little in the way of production start-up costs, so hold the more serious product testing until production product is available. Third, The IBlM P1C Jr. required new learning (always dangerous) and the decision would in turn drive several other key actions. Each of those situations could have been changed slightly to reverse the evaluation dicta. For example, if the coating technology were being bought from Du Pont on an upstream coupling basis, there would be future flexibility, and thus less need for decisive data at this time. If the vitamin line were crowded and required extensive modification and additions.. Figure 1 shows the generally accepted curve (A) for cumulative costs of product innovation, based on some early work by Booz Allen &. Hamilton. The upsweep is gradual because it is an average. However, individual firms often face a curve such as (B) or (C). It varies by project, depending on technical accomplishment required, facilities needed, and so forth. The marketing strategy (e.g. mode 8

FIGURE 1 Cumulative Expenditures Curve Typical New Product and Two Variations Percent of Expenditures (Cumulative) 100 90 80 70 60 50 40 30 20 10 O0 0 10 20 30 40 50 60 70 80 90 100 LAUNCH Percent of Time Spent Figure reads: When X percent of the time has been spent on a new product, Y percent of the total expenditures will have been accumulated. Source: Hypothetical representation, based on empirical curve in Booz, Allen & Hamilton, Management of New Products, (Chicago, 1968), p. 10.

of initial announcement or type of roll-out) is a determinant. The consequence is obvious -- the evaluation done during the progress of one expenditure curve will b e quite different from that of another- A pharmaceutical firm usually has the (B) type curve, meaning that ascertaining medical need and doing a complete market analysis up front are critical. But a food company which can get to prototype fairly quickly and inexpensively will do its serious concept testing with prototypes, not prior to work in the development kitchen. Automobile firms spend so much money introducing their products that they are now working to come up with new methods of simulating test markets - for that last minute, pre-marketing evaluation. Some people feel risk analysis is appropriate here. Any decision involves alternatives, and the various pieces of information to help make that decision have associated costs. The more riding on the decision, the more good data are worth. Eut the opposite is true too -- if nothing big follows, don't belabor the evaluation now. The result of this line of reasoning is shown in Figure 2 which depicts several decay curves. A decay curve is the pattern of project/concept rejection as work proceeds. Some firms spend early, so they need something close to Curve C -- rapid decay, stopping all but highly likely projects. Once they get well down the road, few projects are stopped. Other firms spend late, so they will have an A Curve. Many concepts survive early rounds, but drop like flies when budgets shoot up. Again, there is nothing sacred about any particular decay. And if a firm's new product work is a customary mix of different 1 C)

FIGURE 2 Mortality of New Produc't Ideas Number of Ideas (Percent) 100 V xConception --- Development --- Marketing - Figure reads as follows: At any point of time in the development process, there are Y percent of the original number of ideas left from the number it takes to develop one for marketing. Source: Hypothetical representation based on empirical curve in Booz, Allen and Hamilton, Management of New Products (Chicago, 1968), p.9.

types of projects, there should be a mix of decay patterns. The only problem here is that it is easier for us to talk about upcoming expenditures than it is for a firm to measure them. The reason is opportunity costs. One cost of going ahead is the loss of time on another project which would have been activated or accelerated had we dropped the present one. This is usually highly intangible, tough to estimate, and peppered with political overtones. NEWER EVALUATION TECHNIQUES ARE INCREMENTAL ONES A few researchers are currently trying to create total overall evaluation systems, with a mathematical model for each stage. However, for the most part, today' s evaluation is developing incremental testing approaches. Whereas in the past we strove to perfect the test market to the point where it could do the necessary market testing, today we have at least a half-dozen versions of test market. each designed to meet the needs of developers in different situations. Two of the most dramatic innovations in consumer product market testing in recent years, in fact, cover only portions of what we have known as test marketing. One is the pre-test-market simulation model, in which a simplified mall-setting approach gathers consumers" reactions to a new product and its positioning 'for less than a tenth of the cost of a test market. Some names of services here are BASES, ASSESSOR. COMPI' and LTM. The other new market testing approach is the 'total-city minimarket (or controlled testing) approach where the research firm lines up measurement and/or control over promotional activities (via cable and newspaper arrangements) and over store buying (via arran 12

gements with scanner-equipped supermarkets. Ad Tel, BehaviorScan and Testsight are leaders in this. The prime value of these new methodologies to the new products manager is that they permit buying information about some of the marketing situation without paying for all of it. If a firm knows it can expect pretty good retail distribution on a new item, there is no reason to research the stocking-in process. If product use testing has demonstrated a strong consumer acceptance of an item, once it is used, then further testing can concentrate on the aware/trial dyad. Some other partial evaluation steps are early concept testing, early prototype testing, employee product placement testing, and gradual market roll-outs. No single one of these may be definitive, nor need it be. The manager is building a case, and lots of little tests make more sense than one big test, particularly for a development process in which the product itself is never set until it succeeds in the marketplace. In addition to this effort to pinpoint just what needs to be evaluated, product developers are also moving to a more systematic structure of surrogates. They substitute something that can be measured for something that cannot be. Managers have long done this at the screening stage -- most screening list questions are surrogates. Does the technology fit our manufacturing set-up, are special people skills required, are there complex chemical problems here, etc? The real question -- can we manufacture this at a competitive cost? - cannot be answered directly. Even new product strategies answer this manufacturing cost question indirectly -- e.g. the decision of a canner to stay with

foods that can be canned, or the metal shaping firm to stay with low technology appplications. The point is, we are now systematically studying the development process as a continuous one, seeking to tailor a series of evaluations which meet the real needs at each point, and nothing more. The old idea of "product test. market test, and go" is dead. Figure 3 gives the barebones outline of an evaluation system for a hypothetical new products program. Note that every step is keyed to that particular program. Note, too, that evaluation begins well before ideation and continues well after marketing, with lots of overlapping in the sequence. IMPLICATIONS FOR TODAY' S NEW PRODUCTS MANAGERS By now the reader has thought of ways these concepts can be applied to his/her particular company situation. Here are some general applications: Discard company-wide or division-wide evaluation systems, and demand that every new products team stipulate the specific system it will use for the assignment at hand, as early as possible, preferrably before ideation begins. > Try very hard to avoid GO/NO-GO situations. Break them down. Dilute the focus and the drama. > Put firm new product strategies into place. A product innovation charter, as it were, for every new product program, covering the focus (technology, user target, application), specific goals of the program, and any constraints deemed essential, such as whether to be first, second, or whatever, to market, and which function of the firm probably will have to be the driver. 14

I FIGURE 3 HYPOTHETICAL EVALUATION SYSTEM FOR A NEW PRODUCT PROGRAM PHASE 1: PRECONCEPTUALIZATION: Evaluate the firm, its people, its technical capabilities, etc. Evaluate the marketplace. Spell out product innovation charter, narrow the focus, evaluate innovation options, communicate, and instruct. Generally speaking we will use technological focus, commitment to true innovation, and early market precedence. PHASE 2: CONCEPT TESTING: As ideation begins, use concept testing appropriate to each concept. Concepts are pre-technology, and relate primarily to customer applications. No R&D until concept has support of the marketplace. Exception: if low-cost prototype needed for the evaluation. PHASE 3: SCREENING: Depending on predicted technical cost accumulation, screening may come early or late. The screen will predict the liklihood of our technical accomplishment (both in R&D and in manufacturing), and the liklihood of successful marketing. Screening ends with a statement of the performance attributes required of the product, agreed to by all parties. PHASE 4: EARLY PRODUCT TESTING: As concepts begin to take technical form, there will be early in-house and then customer application. Meanwhile, more market analysis will be done on those concepts that give early positive feedback. PHASE 5: FIELD PRODUCT TESTING: Phase 4 work merges into Phase 5, where we ascertain whether the item has the required performance attributes agreed to at the screening stage. This testing is to be of impartial, typical users, from the target market. During this testing, verify product positioning statements. PHASE 6: MARKET TESTING: Put the product and the marketing plan together for the first time. Interview potential customers to get their opinions and attitudes on our new product, set in a commercial (selling) mode. Then begin offering it for sale; delivery may be delayed until there are sufficient purchases to warrant production. Start with friendly customers, especially those we have talked to earlier, and then move to more neutral ones. Resolve pre-agreed critical issues, usually (1) understanding of the final product concept, (2) willingness to lay out the money for a trial supply, and (3) mode and success of uses. PHASE 7: ROLLOUT: Unless regular customers won't wait, market introduction will be by rollout: regional, customer by customer, or application by application, whichever appears appropriate. Rollout will generally take six months to a year, and will speed up as early indication of success is achieved. Track precise target on concept acceptance, trial rate, and successful application. L II I ~ ~ I Il I

> Organize a multi-functional team for each major new products program, and see that the members operate as a team. Since the very concept of a matrix is to have shared obligation between function and program, matrix teams are only a last ditch option here. > Erase all lines between technical and marketing, and see that any and all evaluations are done by both groups in cooperation. Never allow one side of this pair to make important evaluation-based decisions without the other one's knowledge. > Allocate more dollars to early-on evaluation. Research has shown that over 20 percent of the total costs of the average new product's development were spent prior to R&D. Strategic planning and early concept testing are absolutely vital. > Be sure that marketing strategy for a new product takes form early -- beginning before technical work. True, it will often change, as will product form, and much of the strategy cannot be known early. But what can be k.nown, what is desired and perhaps even assured, should be in writing. > Don't let participants, particularly upper managements, get in the habit of seeking the "true" situation, "actual" profits, "accurate" market feedback. Accept today's evaluations for what they are: estimates of what we will be able to do, later, in a changing environment, with the advantage of much in-formation yet to surface, and with the options to change much of what we currently plan to do. > Above all, don't let thinking come to closure prior to marketing the new product -- it is still "just a concept, in temporary physical form". And it will probably still be temporary until well after marketing. 16