Faculty Research

Differentiated Bayesian Conjoint Choice Designs Zsolt Sandorl Erasmus University Rotterdam and Michel Wedel2 University of Michigan ABSTRACT Previous conjoint choice design construction procedures have produced a single design that is administered to all subjects. This paper proposes to construct a limited set of different designs. The designs are constructed in a Bayesian fashion, taking into account prior uncertainty about the parameter values. A computational procedure is developed that enables fast and easy implementation in practice. Even though the number of such different designs in the optimal set is small, it is demonstrated through a Monte Carlo study that substantial gains in efficiency are achieved over aggregate designs. 1 Postdoc Researcher, Econometric Institute, Erasmus University Rotterdam, P.O. Box 1738, 3000 DR Rotterdam, The Netherlands, E-mail: sandor@few.eur.nl. 2 Dwight F. Benton Professor of Marketing, University of Michigan Business School, 701 Tappan Street, 48109 -1234 MI, Ann Arbor, USA, E-mail: Wedel@umich.edu. 1

INTRODUCTION After its introduction by Louviere and Woodworth (1983), amongst others due to the wide availability of software (http://www.SawtoothSoftware.com), choice experiments have become one of the preferred tools to collect information on consumers' preference structures in fundamental and applied research. With the availability of sophisticated models for the analysis of the collected choice-data, academic interest seems to have shifted to the design of the choice experiments as a topic of primary interest, not in the last place because of the large gains that accrue from proper choice design. The work on conjoint choice design in marketing started with a.o. Kuhfeld, Tobias and Garatt (1994), Lazari and Anderson (1994) and Huber and Zwerina (1996). These authors showed how to construct a design matrix that maximizes the information on the parameters of the Multinomial Logit choice model. They proposed design construction methods that use heuristic searches over the design space under suitable constraints, providing improved, rather than strictly optimal designs. Kanninen (2002) recently showed that D-optimal designs for multinomial choice experiments, where all attributes are quantitative, can also be derived. Sandor and Wedel (2001) demonstrated that choice designs that provide more efficient parameter estimates, also improve predictive validity, a key measure of the effectiveness of conjoint choice studies. Choice designs with improved efficiency minimize the burden on respondents, and reduce the effective sample size needed. But, the optimization of the design is complicated by the fact that the information on the parameters depends on the unknown values of those parameters: a circular problem. Initially, the authors cited above, have resolved that problem by fixing the parameters to certain plausible values when constructing the design. Sandor and Wedel (2001) proposed to alleviate the circular design construction problem by eliciting prior information (from management) and using Bayesian design methods that integrate the optimality criterion over that prior distribution. Even for fairly uninformative priors their 2

procedure was shown to improve over the classical design generating procedures. Arora and Huber (2002) alleviate the circularity problem by first estimating a Hierarchical Bayes model on a pilot sample to obtain initial parameter estimates, and then constructing a more efficient design based on those. Although their choice model deals with individual differences, their design does not, i.e. it is the same for all subjects. Sandor and Wedel (2002) develop designs with improved efficiency for heterogeneous Mixed Logit models that accommodate individual differences. Although it accommodates differences in the logit-model parameters among subjects, again, this method produces a single design to be administered to all subjects. Thus, although much progress in construction of experimental choice designs has been made, designs generated with each of these previous procedures share the drawback that they are aggregate: by applying the procedure in question, one single design is constructed that is administered to every subject in the sample. Against this background, this paper will make a case for the construction of differentiated designs, i.e. design-sets that comprise of several sub-designs. Following Sandor and Wedel (2001), we use a Bayesian design construction procedure that involves the specification of a prior distribution of the parameters. We show that, even with a limited number of sub-designs in the design set, substantial improvements in efficiency over aggregate Bayesian designs can be achieved. We proceed as follows. First, we review the construction of Bayesian designs, then we lay out the procedure for constructing differentiated design sets, focusing subsequently on computational issues that make its implementation in practice fast and easy. After that we present the results of a synthetic data study that compares the differentiated Bayesian design to its closest contender, the aggregate Bayesian design, in a variety of design classes. We end with summarizing the findings, providing simple guidelines for implementation in practice, and discussing avenues for further research and development. 3

DIFFERENTIATED CHOICE DESIGNS Bayesian Designs We start from a conjoint choice model, with the stacked design matrix X = [xj,= ]j...,, where x1, is a k-vector of the attributes of profilej in choice set s. If the S=l,...,S utility of a subject for that profile is uj, = X', P3 + ~ j, where f is a k-vector parameter, and ej,s is an i.i.d. extreme value error term, then the multinomial logit probability thatj is chosen is pji = exp(x 3)/ exp(xs 13). The information matrix is obtained as the variance of the r=1 first order derivatives of the multinomial log-likelihood with respect to the parameters: s (1) I( X)=N. XS(Ps - p spS)X, s=1 where Xs = [xi,s... xJ,s]A ps = [pi,s... PJ,s]' P = diag(pl,...., pJs) and N is the number of respondents. The information matrix plays a crucial role in the construction of choice designs that yield more efficient estimates of the parameters of the MNL choice model. An often-used one-dimensional measure of the efficiency of a choice design is the Dp-error: (2) DP = det[I( I X)]-k. The power 1/k normalizes the determinant, making it proportional to the number of respondents. To circumvent the circularity problem that parameter values need to be available apriori to construct the design, which occurs because the information matrix (1) is a function of the unknown parameters, Sandor and Wedel (2001) proposed a Bayesian approach to construct the designs. It involves specifying a prior distribution of the coefficients, n7 (3 ), that can be informative or uninformative, depending on the availability of prior information. They 4

obtained the optimal design as the one that minimizes the DB-error, that is, the expectation of the Dp-error over the prior distribution of the parameter values: (3) DB = detlI(3 )-i/k ( )d Rk Computationally, the expected information is approximated by drawing r, r= 1,...,R times from its prior, most conveniently the Normal distribution, 7 (P | P0,Eo ), and computing DlR (4) DB (X) detI( X)/ R. r=l The design that provides the most information, and satisfies minimal level overlap was obtained through swapping (Huber and Zwerina 1996) and cycling (Sandor and Wedel 2001). Swapping involves switching two attribute levels among alternatives within a choice set; Cycling is a combination of cyclically rotating the levels of an attribute and swapping them. The study by Sandor and Wedel (2001) revealed that the Bayesian designs are substantially more efficient than standard designs over a wide range of parameter values. Differentiated Design Sets We argue that we can further improve the efficiency of the parameter estimates obtained from a choice experiment substantially, if we differentiate the design. That is, instead of using a single choice design, as has been done in the literature so far, we use several different designs, constructed in conjunction. Because each respondent receives only a single choice design, the burden on each of them is the same as in an aggregate design. The gain in parameter efficiency accrues from different respondents being given different designs, which causes more variation in the choice attributes (the explanatory variables), and enables the variation in the dependent variable to be better captured. Therefore, the estimates of the parameters obtained from differentiated designs are expected to be more efficient, even 5

without prior information that distinguishes the subjects. This is particularly important, since there is no need to collect data on the subjects in the sample before constructing the design. We provide a formal argument. Assume we use a differentiated design-set and respondent i is given the design Xi, for i = 1,...,N. The optimal differentiated design-set can be obtained by minimizing the DB-error over the N designs, that is, (5) min DB(X1,,XN).I N,..., where R (6) DB(Xl,...,XN)=- detlI( X,..., XN /R. r=l Note that the aggregate Bayesian design is obtained by minimizing DB (X) with respect to X as in (3), which is the same as minimizing DB(X1,...,XN) in (6) with respect to X1,...,XN under the restriction Xi =... = XN. This follows since DB(X,...,X) DB(X)N, which holds as a consequence of the information matrix being additive: (7) A(P I x,... )XN)=I(3 I x|)+...+I(3 I XN). Differentiated designs are more efficient because they are the outcome of the unrestricted optimization (5), which necessarily yields a value of the criterion value at least as small as that of the restricted optimization. Computational Issues Minimizing the criterion (5) is a computationally demanding task since N different designs need to be determined, which necessitates searching a very high dimensional design space, since the sample size, N, will usually be large. But, we do not wish to assume that prior information on the parameters of each subject in the sample is available since that would necessitate, for example, a potentially costly pilot study (cf. Arora and Huber 2002). In 6

absence of such prior information, the subjects in our sample are exchangeable, which causes the marginal efficiency of the design-set to decrease as the number of different designs in the set increases. Therefore a design set with a moderate number of designs, X,...,XM, for M < N, may yield close to the same efficiency of the full design-set. We investigate this in our Monte Carlo studies below and show that M= 5 suffices in most cases in practice. But, even forM <N, minimizing the criterion (5) may be computationally demanding, since the M different designs need to be determined simultaneously. In similar complex optimization problems often a so-called "greedy" approach is followed, which in our optimal design problem would involve determining the M designs sequentially. Here we first minimize DB(X) to obtain X1, then we minimize DB (X2 I X ) with respect to X2 to obtain X2, minimizing each design DB(X, I XI,...,X9_ ) for ~=2,...,M, by making use of the previously determined designs. Note that the order in which the greedy algorithm processes the individual designs is arbitrary, since subjects are exchangeable. The sequential approach is still rather demanding computationally. Therefore we propose a simpler procedure that is based on two approximations, which dramatically reduce the computational burden. It makes the construction of differentiated designs even less time consuming than the construction of a single Bayesian design and renders it easy to apply. The first computational gain can be achieved by further simplifying the "greedy" approach with a separate optimization for each of the designs in the differentiated design set. Thus, we construct each design Xi by minimizing DB (X), using different random draws for each design in the construction. This procedure in fact determines designs that are locally optimal, and because these designs are different, they are expected to have good simultaneous performance. Several Monte Carlo experiments (not reported here) show that the differentiated designs obtained by this procedure are very close to as efficient as those obtained by the "greedy" approach, while their construction is much less computationally 7

intensive. This is because the objective function evaluations in the separate optimization of the differentiated designs require significantly fewer operations even for a small number of designs. The second computational issue involves the number of random draws employed in the approximation of the integrals in (3) in the design construction. The number of random draws, R, should enable a sufficiently precise estimate of the objective function DB(Xi). Sandor and Wedel (2001) used R = 1,000 draws to construct their Bayesian designs. Monte Carlo experiments (not reported here) revealed that the efficiency of the differentiated designs did not strongly depend on the number of draws used for their construction. Thisisi so, because in Bayesian design construction, the larger the number of parameter values, the more room for efficiency improvements. Constructing a differentiated design-set requires fewer draws of the parameter values, because the different designs supply extra information. Therefore the number of random draws used to compute the criterion DB(Xi) in the construction does not strongly affect the efficiency of the resulting differentiated design-sets. Our Monte Carlo experiments revealed that the number of draws should be R = 5 x k, roughly five times the dimension of the information matrix. These two computational heuristics lead to a dramatic reduction of computing time needed for constructing the differentiated design-sets. On average, construction of a set of M = 5 or 6 differentiated designs takes only about 25-30% of the time needed for constructing a Bayesian design. But, at the same time they are substantially more efficient than aggregate Bayesian designs, as we show in the Monte Carlo studies below. 8

MONTE CARLO COMPARISON Details on Design of the Study We compare the differentiated design sets to Bayesian designs (Sandor and Wedel 2001), since these have been shown to provide higher efficiency than standard designs for a wide range of true parameter values. In line with previous studies (Huber and Zwerina 1996, Sandor and Wedel 2001), we use four design classes that enable us to investigate the effect of the number of attributes and their levels on the efficiency of the designs. These design classes differ with respect to the number of attributes, three and five, and the number of attribute levels, three and four. The designs with three attributes have twelve choice sets, and the designs with five attributes have eighteen choice sets. All choice sets have two alternatives. We construct three Bayesian designs and three sets of differentiated designs in each of the four design classes, with r7 (3 | Po 0, ) is Normal with O = a0Ik. We set so = 0.20, 1.00, or 2.00, reflecting different levels of uncertainty about the parameter values. We repeat all the computations for the number of designs in the differentiated design sets varying from M = 2 to M= 15. Thus, our study comprises a 22 x 3 x 14 full factorial design with 168 conditions. We use a starting design with minimum level overlap and level balance (Huber and Zwerina 1996), and improve it by applying first the swapping and then the cycling procedures. This way the constructed designs satisfy the minimum level overlap property. For constructing the Bayesian designs we use 1,000 draws. For constructing the differentiated designs, we take the number of draws to be roughly equal to five times the dimension of the information matrix, so that we only need to use 25 draws for the differentiated designs with 3 attributes and 3 levels, 50 draws for the designs with 3 attributes and 4 levels and for the designs with 5 attributes and 3 levels, and 75 draws for the designs with 5 attributes and 4 levels. 9

Then, for each of the 168 conditions specified above, we draw 1,000 true parameter vectors from the normal distribution, with a mean Po and a standard deviation, o, for each of 16 grid points between G = 0 and G = 3. At each of these 16,000 draws we evaluate the Dp-errors of the Bayesian and the differentiated design. Measure of Efficiency The comparison of the efficiency of the Bayesian and differentiated designs is based on the percentage by which the number of respondents for the differentiated designs can be reduced in order to obtain the same efficiency as with the Bayesian design. This measure is easy to interpret, and derived directly as the ratio of the scaled determinants of the information matrices in equations (4) and (6). If positive, it reflects by what percentage we can reduce the number of respondents evaluating the differentiated design-set in order to obtain estimates that are as efficient as those obtained using the Bayesian design; if it is negative it shows by what percentage we should reduce the number of respondents for the Bayesian design in order to obtain the same efficiency as with the differentiated design. Subsequently, we graph the efficiency of the differentiated designs versus that of the standard Bayesian design, against the value of the standard deviation, a, for each condition in the study. This produces a graph of the relative efficiency of the differentiated versus the aggregate design against the extent to which the true parameter values vary from the ones assumed in the design (cf. Sandor and Wedel 2001, 2002) for all 168 conditions. When evaluating the relative efficiencies we take the different numbers of designs explicitly into account so that the obtained relative efficiencies are practically relevant. Results of the Monte Carlo Study 10

The results are presented in Figures 1 and 2, each having six panels. Figure 1 shows the results for designs with three attributes and Figure 2 the results with five attributes. The left hand panels in both figures refer to the three-level case and the right-hand panels to the four level case. The top, middle and bottom panels present the results for the designs constructed with so = 0.20, 1.00, and 2.00, respectively, reflecting different assumptions on the prior assumption on the parameter uncertainty. Each graph shows seven lines, each line comparing the Bayesian design to a differentiated design set with an even number of designs, i.e. M= 2, 4,...,14. We omit the design sets with odd numbers of designs to prevent cluttering of the graphs. [INSERT FIGURES 1 AND 2 HERE] Figure 1 shows that if we compare the efficiency of three-attribute, three-level choice designs for different prior variances, the improvements over the aggregate Bayesian design range from 5 to 55%, if the prior variance is assumed to be small, i.e. the so = 0.20 case. The improvement over the aggregate Bayesian design is smallest, 5-15%, if the true parameters are close to the ones assumed in the design, i.e. for o in between 0 and 1. But, even if we assume this high level of prior certainty, the differentiated design is substantially more efficient than the aggregate if the true parameters are far from the assumed ones: for o = 2.0 -3.0 the improvement is 35-55%. While a design set with four designs is somewhat more efficient than a two-design set, after five or six designs the improvement is negligible. The difference between differentiated design sets with different numbers of designs increases when designs are constructed with larger assumed prior variances, i.e. so = 1.00, or 2.00. Here too, the marginal improvement after five or six designs is quite small. The improvement over the aggregate Bayesian designs is more modest at those higher levels of assumed prior 11

variance, which is likely to be caused by the fact that then the standard Bayesian designs become more efficient (Sandor and Wedel 2001). Still, for design sets with M larger than five or six, the improvements range from 0-10% if the true parameters are close to the assumed ones (G between 0 and 1) and 35-55% if they are relatively far off (G ranging from 2 to 3). The three-attribute, four-level design results in the graphs in the right hand column of Figure 1 reveal that if the attributes have more levels (4 instead of 3), the differentiated designs become even more efficient. Here, if the true parameters are relatively far from the ones assumed in the design, efficiency may be as much as 70-80% higher than for the aggregate design. Apparently, a larger number of levels increase the latitude for improvement of the differentiated design. For more than five or six designs in the design set, efficiency does not seem to be much better. For those designs with M > 5, performance is not affected very much by the prior variance assumed in constructing the design, except when the true values are quite close to the assumed ones. The five-attribute three-level designs in the left part of Figure 2 show comparable relative performance over the aggregate Bayesian design; increasing the number of attributes in the design offers a fairly similar degree of design improvement as increasing the number of levels per attribute. If the true parameters are close to the ones assumed in constructing the design (G = 0.5-1.0), improvements over the aggregate Bayesian design are modest, 0-25%, irrespective of the assumed prior variance. But, if the true parameters are farther from the assumed ones (G = 2.0-3.0) efficiency is substantially higher, between 60 and 80%. The marginal improvement of differentiated design sets with more than five or six designs again is small. Improvements over standard Bayesian designs are fairly similar for different values of the assumed variance, except perhaps if the true parameters are in the vicinity of the assumed ones. 12

For the five-attribute four-level designs in the right hand column of Figure 2, the improvements of the differentiated designs over the standard Bayesian designs are even more impressive, ranging from 10% to over 90%, caused by the fact that the higher numbers of attributes and levels both result in much additional room for constructing better designs. Substantial increases in efficiency of 20-30% are realized even if the true parameters are fairly close to the assumed ones, i.e. o = 0.5-1.0. Differentiated designs constructed with a larger prior variance have higher relative efficiency than Bayesian designs, in particular when the true and assumed parameter values are close. Differences between design sets with different numbers of designs are substantial, but again level off after five or six designs in a set, which seems a fairly robust finding across all conditions in our study. Tables 1 and 2 illustrate two differentiated design sets with M = 5, namely, those from the two five-attribute classes constructed using the prior variance so = 1.00. We opted to present these design classes because we find them most relevant for practical purposes. The cumulative DB-errors are also presented. Similar to Figures 1 and 2 these enable a direct comparison in terms of practical usefulness of the designs. That is, the DB-error of the first design (the leftmost column) is multiplied by 1/5, the value presented for the second design is the DB-error of the first and second designs multiplied by 2/5, and so on. This way these DBerrors correspond to situations where the number of respondents is the same irrespective of the number of designs used. As we expect, on the basis of the arguments presented below equation (5) as well as the results from Figures 1 and 2, the DB-errors decrease as the number of designs increases, revealing that the total differentiated design set yields more and more efficient parameter estimates, or equivalently, requires less and less subjects to achieve the same efficiency. [INSERT TABLES 1 AND 2 HERE] 13

CONCLUSION We believe that our study presents a strong case for constructing differentiated, rather than aggregate (Bayesian) designs. All prior studies on conjoint choice design construction, even when dealing with models that account for heterogeneity (Arora and Huber 2002) or constructing designs that are more efficient when the sample is heterogeneous (Sandor and Wedel 2002), have developed a single design that is to be submitted to all subjects in the study when collecting the choice data. This study demonstrates that constructing several choice designs, and distributing the randomly across subjects, even in the absence of information that distinguishes the subjects a-priori, or assuming that the sample is heterogeneous, yields substantially higher efficiency. It is important to note that the use of the differentiated design sets comes at no additional cost or response burden to subjects. Our results show, first of all, that if we have little uncertainty about the values of the parameters in a conjoint choice experiment, and the true values are indeed close to the ones assumed in the design, then, a differentiated design offers moderate improvement over aggregate (Bayesian) designs. Sandor and Wedel (2001) already showed that in that case there is not much difference in efficiency between aggregate Bayesian and standard design generating procedures, so that all procedures in that case yield designs with similar efficiency. This finding is intuitive, since if we know the parameter values with reasonable certainty, there is not much use in collecting additional data, and all design generating procedures should provide similar choice designs. Nevertheless, this result may be important for studies that set out to replicate previous findings. However, if we are not certain about the parameter values, and we assume a large prior variance in constructing the designs to reflect that uncertainty, differentiated designs produce substantial improvements in efficiency over aggregate designs. We consider this of primary 14

interest, since in most cases when conducting a conjoint study the purpose is to estimate parameters that we have no or little prior knowledge about. It turns out that if we assume a large prior variance, and the true parameters are far from the point estimates we assumed in constructing the design, the differentiated designs are much better than the aggregate (Bayesian) designs, since they produce a higher spread of the design points which enables more precise estimation of the parameters. The improvements in efficiency of the differentiated designs over the aggregate designs increase with both the number of attributes and with the number of their levels. Such designs provide more latitude for improvement, which is important in applied work, since there large designs with many attributes and many levels seem to be prevalent. Thus, the concept of differentiated designs is likely to be of value especially in applied studies that involve a large number of attributes at many levels. Our Monte Carlo study provides a clear guide as to the number of designs needed in a differentiated design set. Across all conditions in the study, it appeared that after five (at most six) designs in the design set the improvements are negligible. In addition, based on the results, we suggest that unless one is quite certain about the true parameter values, for example from a previous study among a random sample from the same population as in Arora and Huber (2002), the prior variance can be set to so = 1.00 in constructing the a differentiated design, since in most cases the effect of setting a larger prior variance is negligible. These general findings, yielding preferred settings of M = 5 and so = 1.00, along with the speed of computation due to the small number of draws and independent optimization of the different designs in the set, makes our procedure easy to implement in practice. For future research, the greedy algorithm we have developed holds promise for sequential design construction. This holds in particular for on line choice data collection. After a first design has been constructed and administered to a sample of subjects, a second 15

design may be constructed, conditional upon the first design, and potentially the responses of the subjects interviewed thus far. The speed of computation of the design generating procedure makes real-time computations involved feasible. A further avenue for future research is to tailor the design to prior individual level information, which again may be of use in online choice data collection. Here, the design can be updated in a Bayesian fashion as new choice information on the subject comes available. Such a procedure would enable a choice-based version of adaptive conjoint analysis, where the choice design itself is adapted sequentially. Since collection of prior information and the optimization based on it need much more work, we leave these issues for future research, but consider the current study as an important first step in pinning down the concept of differentiated designs and demonstrating the substantial improvement in choice model parameter estimation that they result in. 16

REFERENCES Arora, Neeraj and Joel Huber (2001), "Improving Parameter Estimates and Model Prediction by Aggregate Customization in Choice Experiments," Journal of Consumer Research, 28 (2) 273-283. Huber, Joel and Klaus Zwerina (1996), "The importance of Utility Balance in Efficient Choice Design," Journal of Marketing Research, 33 (August), 307-17. Kanninen, Barbara J. (2002), "Optimal Designs for Multinomial Choice Experiments," Journal of Marketing Research, 39 (May), 214-227. Kuhfeld, Warren F., Randall D. Tobias and Mark Garratt (1994), "Efficient Experimental Design With Marketing Research Applications," Journal of Marketing Research, 31 (November), 545-57. Lazari, Andreas G. and Donald A. Anderson (1994), "Designs of Discrete Choice Experiments for Estimating Both Attribute and Availability Cross Effects," Journal of Marketing Research, 31 (August), 375-83. Louviere, Jordan J. and G. Woodworth (1983), "Design and Analysis of Simulated Consumer Choice or Allocation Experiments: An Approach Based on Aggregate Data," Journal of Marketing Research, 20 (November), 350-367. Sandor, Zsolt and Michel Wedel (2001), "Designing Conjoint Choice Experiments Using Managers' Prior Beliefs," Journal of Marketing Research, 38 (November), 430-444. Sandor, Zsolt and Michel Wedel (2002), "Profile Construction in Experimental Choice Designs for Mixed Logit Models," Marketing Science, 21 (4), 55-75. 17

Figure 1: Efficiency Comparisons of Bayesian and Differentiated Three-Attribute Designs for Three Levels (Left Hand Side Panels) and Four Levels (Right Hand Side Panels)

c0 U C o CT o 0) (DU 0 C 0 rl -2 designs - 4 designs..... 6 designs -- 8 designs --— 10 designs — 12 designs -14 designs i I i I i I i I I I i (0 C) _0 ). - — 2 designs -10 designs C -.. 12 designs 0 — 14 designs 1I I0 0.5 1 1.5 2 2.5 3 0 0.5 1 1.5 2 2.5 3 Deviation of True from Assumed Mean Deviation of True from Assumed Mean c0 o o '(o m w o CD U ) oc L o U(D 0 N (V -2 designs - 4 designs -.. 6 designs -- 8 designs --— 10 designs — 12 designs -14 designs 0 0.5 1 1.5 2 2.5 3 c 0 Tj 0 0) U) -P C c (D L U) I 0 o 00.. -2 designs r. ~:' 4- 4 designs....... 6 designs -- 8 designs ^^^ ^ ""'""' 1 ---10 designs -. 12 designs ol -14 designs I0 0.5 1 1.5 2 2.5 3 Deviation of True from Assumed Mean Deviation of True from Assumed Mean Ci 0( C( -0 U) 0)( 0) Ela U) Lo 0Q. LC 0.5 1 1.5 2 2.5 Deviation of True from Assumed Mean Deviation of True from Assumed Mean

Figure 2: Efficiency Comparisons of Bayesian and Differentiated Five-Attribute Designs for Three Levels (Left Hand Side Panels) and Four Levels (Right Hand Side Panels)

-o 0c( Th L)C _0 U) 0. -o U) U) 0) U) -o U) Deviation of True from Assumed Mean Deviation of True from Assumed Mean Deviation of True from Assumed Mean Deviation of True from Assumed Mean 0.5 1 1.5 2 2.5 3 ' 0 0.5 1 1.5 2 2.5 Deviation of True from Assumed Mean Deviation of True from Assumed Mean

Table 1: The First Five Differentiated Designs in the Class 35/2/18 1St design 2nd design 3rd design 4th design 5th design Choice Profile Attributes Attributes Attributes Attributes Attributes Set 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 I 32112 1 3212 22212 23112 12222 II 23321I 2331 1 333 11221 213 11 2 I 22122 12132 12133 13123 33112 II 3121 312 21 33221 31231 11231 3 I 2222 1 31311 311 1 332 1 11221 II I1111 3 22123 12223 21123 22 112 4 I 21222 11222 23122 13322 31122 II 1 33 13 2211 3 122 1 3 2211 3 12233 5 I 13221 2 2 211 112 21 2 122 2 3 2 2 1 II 2 2113 1112 2 2211 3 1211 3 12113 6 I 31121 12121 2 2 121 13221 13311 II 1222 2 1212 13212 21112 21122 7 I 2111 3 2111 3 1231 3 33 31333 12213 II 1 22 2 232 22 3 2 23121 3 2 21332 8 I 3 1331 12311 12221 2 2 12 1 3 2 2 2 2 II 12 22 13 2 113 3 2131 31 12 12 21113 9 I 11332 12312 13112 22312 3 22212 II32211 31131 22231 31131 31131 10 I 3313 3 2223 31323 11313 2 1323 II 21332 2313 21313 2 32122 123132 11 I 11331 13231 3 113 1 2 3311 2 3 131 II 232 3 122 2 322 11232 12212 12 I 31312 3313 2 12332 12332 1 3 2 3 2 II 1 2 2 3 3 2 1 3 1 3 2 323 2 3213 2 3 2 2 12 3 13 I 3 3 1 3 2 2 11332 13 132 1 3 3 2 2 II 2 2 3 2 3 3 2 1 2 312 I3 3 2 1 3 1 14 I 1 2 3 2 3 1 2 2 3 2 1 3 1 3 1 2 3 2 3 1122II 223 2 2 3 22 1222 3 12 1 2 15 I 3 2 131 3 2 1 1 1 22 21 1 2 3 2 1 II 2 12 2 3 112 2 3 3 11 3 3 3 1 12 3 2 12 3 3 16 I 3 3 3 1 2 3 3 12 2 1 3 12 3 2 2 12 1 3 3 2 I I 11 3 1 2 12 3 3 3 1 3 1 2 3 2 2 2 11 17 I 3 12 2 3 3 2 2 1 3 1 12 2 3 2 12 2 3 3 12 2 3 11 2 2 1 1 2 113 2 1 2 3 3 12 3 1 1 2 2 3 3 1 18 I 1 2 12 2 2 2222 3 2 1 22 3 3 122 1 3 12 2 11 2 1 2 3 3 12 3 3 21233 2223 3 1 2 3 3 DB-error 0.193 0.126 0.116 0.112 0.110

Table 2: The First Five Differentiated Designs in the Class 45/2/18 St design 2nd design 3rddesign 4thdesign 5thdesign Choice Profile Attributes Attributes Attributes Attributes Attributes Set 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 I 12343 2421 3 23211 242 1 3 23212 II 24234 32124 11322 32322 11323 2 I 32141 12231 2133 1 2144 1 41211 II 23423 2 1113 322 13 12323 12333 3 I 31122 1 3 114 31124 11442 21244 1 3I 1 3231 3124 1 13233 3313 1 43331 4 I 22242 11232 33222 41331 32242 II 314 14 22424 244 14 32123 23414 5 I 13314 4131 2 22224 2 3 3 14 42114 II 2 2 4 3 1243 3 3 3 141 12431 1 3 2 3 1 6 II 311 3 31132 232143 31323 1 1432 I 11322 3 31 1 14312 43112 3 3 2 2 3 7 I 312 2 3 11233 2 2123 23413 1 3 2 2 3 II 4 4311 22321 312312 32341 2 2 3 3 1 8 I 1 2 2 4 2 3122 2414 2 32142 1 2 4 2 2 II 3 3 3 2 1 12141 3 3 4 21 1 3 2 21 31143 9 I 23132 4 2 3 3 2 12132 3 3 3 3 2 3 3 3 2 1 II 3 2 2 2 4 1 3 2 2 4 212 2 4 42424 2 2 2 3 3 10 I 3 2 12 22 2213 4 1312 4 3 112 4 21322 II 21313 1 3 3 2 3 4231 3 2231 3 12113 11 I 32212 12212 22232 12232 13132 II 13131 31331 4 3 111 31311 32211 12 I 3231 2 23132 24132 21132 3 2 4 3 2 II 4 4 1 3 3 1313 1231 3 1 3 3 1 3 2 4 2 1 3 13 I 12 112 3 4 1 1 2 1 3 4 2 2 3 2 2 12 3 1 1 12 II 3124 12 4 3 2 4 3 3231 4 12 2 4 1 14 I 4 3 2 2 1 4 3 4 11 4 2 2 4 1 3 4 4 3 1 14 2 3 1 II 12 3 3 3 3 2 12 3 3 3 3 3 3 2 3 14 3 2 114 3 15 I 3 4 2 1 2 4 2 4 2 2 2 1 4 1 2 14 3 2 2 2 14 1 2 11 2 2 1 3 4 14 3 4 4 1 3 1 3 4 2 2 2 4 4 1 3 3 4 16 I 1 3 4 4 2 1 4 4 4 2 2 3 3 22 2 332 3 3 2 2 1 4 3 4 2 II 4 12 3 3 2 2 2 3 31133 311 3 3 2 2 1 3 3 17 I 2 4 12 1 44 12 1 3 242 1 44 12 1 2 4 12 1 11 11 3 3 2 3 3 3 12 2 12 14 1 1 3 3 4 3 1 3 14 18 I 3 1 3 14 2 1 3 12 3 1 3 14 12 1 1 4 1 1 3 14 II 1 2 2 2 3 4 2 2 2 3 1 4 2 2 3 3 12 2 3 3 4 2 2 3 DB-error 0.437 0.217 0.185 0.173 0.167