RESOLVING DILEMMAS THAT OCCUR WHEN USING QUALITY MEASURES Karl D. Majeske' and Richard W. Andrews The University of Michigan Business School 734 Tappan Street Ann Arbor, MI 48109-1234 ABSTRACT Manufacturers and suppliers use quantitative criteria, such as process capability indices, to make informed decisions regarding product quality. Manufacturers also use quantitative criteria, like the precision to tolerance ratio, to approve a gage or measurement system. This research suggests that manufacturers include an additional criterion, the correlation in repeat measurements, when measuring and evaluating product quality. We derive the theoretical correlation in repeat measurements as a function of product and gage standard deviations. By plotting the precision to tolerance ratio and the correlation in repeat measurements on Cartesian coordinates - the axes being sigma gage (X) and sigma part (Y) - it is shown that contradictory decisions can occur regarding the aptness of the measurement system. We then add Cp (a measure of process capability) to the plot. The relationship between the three measures suggests a method for determining an acceptable level for the correlation criterion. The appendix explores the effect of measurement error on the manufacturer's estimate of process capability. 1 The first author was supported in-part by NSF grant SBR-9712997 to The University of Michigan By Karl D. Majeske and Richard W. Andrews July 17, 2001 Page: 1

RESOLVING DILEMMAS THAT OCCUR WHEN USING QUALITY MEASURES I) INTRODUCTION To make objective decisions regarding product and process quality, manufacturers and their suppliers use quantitative quality measures. It is common for manufacturers to approve a supplier's process by setting required values for measures of quality. Consider the case in which a supplier is developing custom tooling, e.g., developing dies for stamping automotive sheet metal components such as doors, hoods and fenders. When making the decision to purchase the tool or die, the so-called tool buy-off, the automobile manufacturer and the tool construction supplier will communicate using quality measures based on dimensional data. As another example, consider the situation in which manufacturers and suppliers use dimensional data as a basis of communication when trying to resolve build problems in assembly. In this paper, we assume a manufacturer requires their supplier's to meet quality P criteria as measured by (1) the precision to tolerance ratio, and (2) the process T' capability measure Cp. The precision to tolerance ratio is a statistic used to make a decision about the acceptability of the measurement system. The capability index Cp shows the potential of the production process to produce products within design specifications. We suggest adding the correlation in repeat measurements p as a third measure in the manufacturer - supplier relationship. This correlation measures the correspondence between the measurement processes. All three measures are important for the breadth they bring when considering quality. We explore the relationships among these three measures and show the importance of understanding these relationships when setting required values. The By Karl D. Majeske and Richard W. Andrews July 17, 2001 Page: 2

RESOLVING DILEMMAS THAT OCCUR WHEN USING QUALITY MEASURES relationships among these three quality measures are shown using both equations and graphs. The graphs plot the measures on Cartesian coordinates with the axes being the standard deviation of the measurement gage (X) and the standard deviation of the quality dimension (Y). The understanding of these relationships will assist quality managers in setting required values. The remainder of this paper has the following organization. Section 2 presents a model for the data generated when measuring a quality characteristic. Section 3 discusses a measurement system acceptance criteria called the precision to tolerance ratio. Section 4 explores the correlation of repeat measurements in the presence of measurement error. Section 5 shows a relationship between correlation and the precision to tolerance ratio. We show that, in some situations, using both criteria can lead to contradictory decisions regarding the measurement system. Section 6 discusses the measure of process capability Cp, develops a relationship between the three quality measures, and recommends how to resolve the dilemmas that may occur when using multiple quality measures. II) MEASURED VALUES Ideally, the measurement system would generate data that exactly represents the geometry of the part. In reality this is not the case. We present the following model for measurements taken on a quality characteristic. Let P represent the true unknown value for the part. We assume that P follows a normal distribution with mean up and variance 2 Op or P - Normal(,u, op ). By Karl D. Majeske and Richard W. Andrews July 17, 2001 Page: 3

RESOL VING DILEMMAS THAT OCCUR WHEN USING QUALITY MEASURES Let G represent the gage error induced by the measuring process. We assume that G follows a normal distribution with variance ac and mean,g or G - Normal(ug, a 2). Lastly, the random variable X represents the measurement. We assume the product dimension P, and the measurement error G, are additive resulting in X=P+G. (1) In an attempt to force accuracy into the readings, manufacturers will calibrate a gage. The act of calibration removes any potential bias in the measurements generated using the gage. Gong, Yuan and Ni (2000) present a method for calibrating an optical measuring device used to measure large automotive sheet metal stampings. By assuming the gage has been calibrated, i.e., assuming ug = 0, we have ix = A p. To determine the variance in the measurements we take the variance of equation 1. By assuming that gage error is statistically independent of the quality characteristic, the variance in the measurements is a2 = -2 + o. (2) This results in X following a normal distribution X - Normal(Atx = / 2tp = a + g ). III) MEASUREMENT SYSTEM ACCEPTANCE CRITERIA A commonly used quantity for assessing the precision of a measurement system is the precision to tolerance ratio. The precision to tolerance ratio applies when the quality By Karl D. Majeske and Richard W. Andrews July 17, 2001 Page: 4

RESOLVING DILEMMAS THAT OCCUR WHEN USING QUALITY MEASURES characteristic has a two-sided design specification or tolerance. Letting USL denote the upper specification limit, and LSL denote the lower specification limit, we formally define the tolerance as: Tolerance = T = USL - LSL. Even though the theoretical width of G is infinite, Montgomery (1997) suggests using 6 standard deviations to express the width of the measurement error distribution and thus calculating the precision to tolerance ratio as p 6crg ~~~- ~~~~~= --- ---.~ ~(2) T USL- LSL Domestic manufacturers in the United States automobile industry (Automobile Industries Action Group 1997) suggest approving a measurement system with p 5.15-g ~-=~~~~-: --- -'~~~~ ~ ~(3) T USL- LSL P The supplier or manufacturer will compare - to some standard for the purpose T of approving the gage. Montgomery (1997) suggests that gages with a - of 0.1 or less T should be adequate. The Automobile Industries Action Group (AIAG) suggests three ranges of their statistic: P 1) -<0.1 the gage is capable, T 2) - > 0.3 the gage is not capable T P 3) 0.1 < - < 0.3 the gage may be capable. T By Karl D. Majeske and Richard W. Andrews July 17, 2001 Page: 5

RESOLVING DILEMMAS THAT OCCUR WHEN USING QUALITYMEASURES Notice that the precision to tolerance method depends only on the variance of the measurement system and the width of the design specifications, but not on the product P variance. In the following sections we will show how - relates to correlation in repeat T measurements and process capability. IV) CORRELATION IN PRODUCT MEAUREMENTS When purchasing stamping dies from their suppliers, automobile manufacturers will have them stamp a set of five to ten panels. The tool construction supplier will measure the stampings and ship them, along with the dimensional data, to the automobile manufacturer. The automobile manufacturer measures the panels again generating a second set of measurements. We have observed situations between a manufacturer and supplier where differences in the mean and variance are statistically insignificant, however, the two sets of measurements have very low correlation. This lack of correlation in the repeat measurements inhibits the two parties from reaching an agreement on the condition of the stamping die. Let X1 and X2 represent the supplier and manufacturer measurements respectively for a given part. The correlation is given by p(X1,X2) = Cov(X1 X2) and can be expressed in terms of gage variance and product variance. First, note that E[X] = E[E[X P]] = E[P], Var[X] = 42 + r2 and, Cov[X1, X2] = E[X1X2 ] - E[X ][X2 ]= E[XX2] - (E[P])2. By Karl D. Majeske and Richard W. Andrews July 17, 2001 Page: 6

RESOLVING DILEMMAS THAT OCCUR WHEN USING QUALITY MEASURES Now E[XiX,2] = E[E[X,,Xi2 1 Pi]] = E[E[X,, I P]E[X2 I Pi]] = E[PiPi] = E[i2 ] so, Cov[X, X2] = E[P2 ] - (E[P])2 = Var[P] = crp and 02 1 p(X X, )=- P = (4) a2 2 2(4) p g 1+ g 2 Equation 4 shows the relationship between the correlation in repeat measurements, p, and the ratio of sigma gage to sigma part. When the ratio is zero (zero measurement error variance) the repeat measurements are perfectly correlated, i.e., the correlation is one. When gage variance equals product variance the correlation is 0.5. Notice that the gage correlation is always greater than zero and that the smaller the gage variance, relative to the part to part variance, the closer correlation is to one. Conversely, the larger gage variance, relative to part to part variance, the smaller the correlation. In other words, a large correlation is bad news for the production process and/or good news for the measurement process, whereas a small correlation is good news for the production process and/or bad news for the measurement process. Of course, both variances are of interest and it is important to keep both of them small. This analysis shows that correlation by itself does not provide all the answers. The correlation criteria should be used with other measures of quality such as the precision to tolerance ratio and process capability measures. By Karl D. Majeske and Richard W. Andrews July 17, 2001 Page: 7

RESOLVING DILEMMAS THAT OCCUR WHEN USING QUALITY MEASURES V) RELATING PRECISION TO TOLERANCE RATIO TO CORRELATION Consider the situation where a manufacturer has determined that a measurement system must possess a precision to tolerance ratio of - < So to be considered appropriate for measuring parts. In addition, the manufacturer requests that the correlation with the supplier's measurements be at least p > po. Notice that for a given go, the supplier needs 0 T g < 6 (5) g 6 to satisfy the - criteria. Also, for a given p0, the manufacturer and supplier need T crp > g Po (6) to satisfy the correlation criteria. Using '0Q = 0.3, po = 0.8, and T = 1 (without loss of generality), we constructed Figure 1 to graphically relate the two criteria in the space defined by sigma gage and sigma part. Gages whose arg is on or to the left of the vertical line would pass the criteria while gages on or above the dotted line would T satisfy the correlation criteria. These two lines then divide Figure 1 into four regions: Region 1: The gage satisfies both criteria. P Region 2: The gage does not satisfy - yet it passes the correlation criteria. T Region 3: The gage doesn't meet either criterion. Region 4: The gage satisfies - but fails to meet the correlation criteria. T By Karl D. Majeske and Richard W. Andrews July 17, 2001 Page: 8

RESOLVING DILEMMAS THAT OCCUR WHEN USING QUALITYMEASURES 0.2 -, Region 2 0.15 Region 1 " E 0), Region 3 0.05 - pO.8 - - Region 4 =.3 0 -I I I 0 0.02 0.04 0.06 0.08 0.1 Sigma Gage P Figure 1: Comparing - and Correlation Gage Approval Criteria T Using two measurement system approval criteria provides a more thorough analysis of the measurement system. However, as we have shown with Figure 1, it creates some possible dilemmas. In regions 2 & 4 the criteria provide opposing P assessments of the measurement system. In region 2, - rejects the gage as not being T precise enough to measure parts, however, the correlation criteria is satisfied. In Region 4, the precision to tolerance ratio accepts the gage but the correlation criteria is not satisfied. It is this phenomenon of a precise gage with a small correlation that initially motivated this research. In section 6 we further explore these anomalies by including process capability as a third quality measure. By Karl D. Majeske and Richard W. Andrews July 17, 2001 Page: 9

RESOLVING DILEMMAS THAT OCCUR WHEN USING QUALITY MEASURES VI) Relating P/T, Process Capability Cp and Correlation p Process capability indices represent a class of quality measures for quantifying process output relative to the tolerance or width of the design specification. Suppliers approve a process for production and delivery using measures of process capability such as Cp and Ck (see Chou, Owen and Borrego 1990, and Franklin and Wasserman 1992). When using measures of process capability, practitioners often assume that the measurement system generates data that they can use to directly estimate the process variance. The Appendix contains a discussion of this assumption and how ignoring measurement error can impact the manufacturer's perception of process capability. The process capability measure T C,- (7) 6ap is the ratio of two widths: the tolerance (T), and the distribution of the quality characteristic (using 6 standard deviations to quantify width). Manufacturers and suppliers approve a process for production if Cp > C. Automobile manufacturers and suppliers that follow Automobile Industries Action Group (1995) guidelines use a cutoff of CO = 1.67, i.e., processes with Cp > 1.67 are approved for production. Notice that for a given C0, the manufacturer or supplier needs T P <C- (8) 6Co In this section, we discuss how manufacturers and suppliers can use the three quality measures in consort to make decisions about their processes and measurement systems. The relationship that ties the measures -, p, and C, is T By Karl D. Majeske and Richard W. Andrews July 17, 2001 Page: 10

RESOLVING DILEMMAS THAT OCCUR WHEN USING QUALITYMEASURES 1 1+ -Cp TT~P (9) Due to this relationship, a manufacturer that wishes consistency in their quality measures should determine acceptable values for - and Cp, and then solve for p using equation T (9) above. For example, an automotive supplier who uses Cp = 1.67 and = 0.3 to T define a capable process and approve a measurement system respectively, should use iO = 0.8 as a cut-off for the correlation criteria. 0.2 tE 0) co 0.15 - 0.1 -0.05 - Region 1a................................................, 'p Region 1b,,, - - Region 4 0. 00 Region 2,-, -, - Region 3a................................................ Region 3b.... - Po = 0.8 65 = 0.3............ Co =1.67 0 I P I I I I I,~~~~~~~~~~~~~~~~~ I 0 0.02 0.04 0.06 0.08 0.1 Sigma Gage P Figure 2: Comparing -, Correlation and Process Capability T I In section 5 we show Figure 1 which plots 80 = 0.3 and p0 = 0.8 in the space defined by sigma gage (X) and sigma part (Y) (using T = 1 without loss of generality). Figure 2 adds the horizontal line representing process capability C0 = 1.67. By using a value of p, that satisfies the relationship of equation (9), we have forced the three lines By Karl D. Majeske and Richard W. Andrews July 17, 2001 Page: 11

RESOLVING DILEMMAS THAT OCCUR WHEN USING QUALITY MEASURES of Figure 2 to intersect. Table 1 shows the six regions of Figure 2 that represent the possible scenarios for a manufacturer or supplier assessing a production process and measurement system. A "yes" indicates that the process or measurement system satisfies the column criteria for the region of the sigma gage - sigma part space. Table 1: Region Classification by Decision Outcome Precision Process Region Correlation to Tolerance Capability la Yes Yes No lb Yes Yes Yes 2 Yes No No 3a No No No 3b No No Yes 4 No Yes Yes From Figure 2 (and previously Figure 1), using correlation and the precision to tolerance ratio to assess a measurement system that falls in either Region 2 or 4 may result in inconsistent conclusions. In region 4, even though the process satisfies the capability criteria and the gage satisfies the precision to tolerance ratio, the correlation constraint fails to be met (the case that gave rise to this research). It seems contradictory to tell a supplier that his process is capable and his measurements meet the precision requirements, but we can't adequately correlate our measurements. This conundrum occurs because the ratio g is too large. Therefore, the more capable a process, the ppp more precise a gage must be to pass the correlation criteria. You can see this graphically in Figure 2. Notice that the width of region 4 increases as sigma part decreases (or Cp increases). By Karl D. Majeske and Richard W. Andrews July 17, 2001 Page: 12

RESOLVING DILEMMAS THAT OCCUR WHEN USING QUALITYMEASURES In region 2, the measurement system satisfies the correlation criterion but does not meet the precision to tolerance ratio. In this situation, we suggest the manufacturer or supplier include process capability when drawing conclusions regarding their measurement system. The manufacturer should evaluate C - knowing that Cp < CO in Region 2 - and make a decision regarding the process. If engineers and managers approve the process for production, it is possible the gage can reproduce readings with high enough correlation to resolve assembly problems and approve design changes. However, if the variation in the process is reduced (making the process capable), without P reducing gage variance, the manufacturer may find that both - and p reject the T measurement system. All three requirements are met in region lb and the manufacturer and supplier should approve both the process and the measurement system. In region 3a none of the three criteria are satisfied. The manufacturer should not approve the process for production and continue trying to reduce measurement error. In region la the manufacturer should conclude they have a good measurement system and a process that is not capable. The benefit of being in region la is the manufacturer has the ability to measure or detect improvement in process capability when such improvement occurs. Region 3b can present a quandary for the manufacturer. Both measurement system criteria suggest the gage should not be used yet the process is capable. Can the manufacturer conclude they have a capable process without acknowledging the measurement system functions? In some cases, manufacturers use dimensional data as a basis for communication. We recommend that manufacturers in this situation add an additional criterion to the By Karl D. Majeske and Richard W. Andrews July 17, 2001 Page: 13

RESOLVING DILEMMAS THAT OCCUR WHEN USING QUALITY MEASURES measurement system analysis process. This criterion represents the correlation in repeat measurements. Using this criterion to approve the measurement system should facilitate communicating via dimensional data by ensuring a specified level of theoretical correlation in repeat measurements. By Karl D. Majeske and Richard W. Andrews July 17, 2001 Page: 14

RESOLVING DILEMMAS THAT OCCUR WHEN USING QUALITY MEASURES REFERENCES Automotive Industries Action Group (AIAG) (1997), Measurement Systems Analysis, Second edition, AIAG, Southfield, MI. Automotive Industries Action Group (AIAG) (1995), Production Part Approval Process, Second edition, AIAG, Southfield, MI. Chou, Y. M., Owen, D. B. and Borrego, S. A., (1990), "Lower Confidence Limits on Process Capability Indices", Journal of Quality Technology 22, No. 3, pp. 223-229. Franklin, L. A. and Wasserman, G. (1992), "Bootstrap Lower Confidence Limits for Capability Indices", Journal of Quality Technology 24, No. 4, pp. 196-210. Gong, C., Jingxia, Y. and Ni, J. (2000), "A Self-Calibration Method for Robotic Measurement System", Journal of Manufacturing Science and Engineering, Transactions of the ASME 122, No. 1, pp. 174-181. Montgomery, D.C. (1997), Statistical Quality Control, Third Edition, John Wiley & Sons, Inc., New York, New York. By Karl D. Majeske and Richard W. Andrews July 17, 2001 Page: 15

RESOLVING DILEMMAS THAT OCCUR WHEN USING QUALITYMEASURES APPENDIX In this appendix we explore the impact of measurement error on the manufacturers assessment of process capability. In this scenario, we assume that the manufacturer uses the sample standard deviation of measured values to estimate u. The sample standard deviation actually estimates a.C (the standard deviation of the measured values) not a, (the standard deviation of the quality characteristic). We will denote the measure of process capability that uses the standard deviation of measured values as Tolerance USL - LSL Cpx 2 (10) PX 6o-, 6o-27 + g Since ap = af -, this approach over-estimates the variability of the process which in turn under-estimates the capability of the process. We directly relate Cp to Cx by P incorporating the precision to tolerance ratio: T Cp =21 (11) i1 j Equation (11) above provides a method for manufacturers and suppliers to determine Cp using the standard deviation of measurements and incorporating the results of a gage capability study. Recall from equation (8) that to approve a process using Cp, the manufacturer T needs ap <. If the manufacturer unknowingly used Cpx, they would need to Co6 satisfy the constraint By Karl D. Majeske and Richard W. Andrews July 17, 2001 Page: 16

RESOLVING DILEMMAS THAT OCCUR WHEN USING QUALITYMEASURES p T < -.2 (12) Using C = Cxo = 1.67 and T = 1 we show Figure 3 to compare the indices Cp and Cp in the space defined by crg and up. A process whose sigma part is below the dotted line would be classified as capable while processes above the line would not meet the capability criteria. Likewise, a process under the curve representing Cpx would be considered capable while a process above the curve would not. The curve and line on Figure 2 divide the space into three regions: Region 1: The process fails to satisfy either criteria Region 2: The process meets the Cp criteria but does not meet the Cpx criteria. Region 3: The process satisfies both criteria 0.16 0.14 - 0.12 - Region 1 0.1 ---- C =- 1.67 ~m 0.08 E 00 -— Co =1.67 C 0.06 0.04 - Region 3 Region 2 0.02 0 0 0.05 0.1 0.15 0.2 Sigma Gage Figure 3: Comparing Cp and Cpx By Karl D. Majeske and Richard W. Andrews July 17, 2001 Page: 17

RESOLVING DILEMMAS THAT OCCUR WHEN USING QUALITY MEASURES In region 1, using either Cp or Cp to quantify a process would lead to the correct conclusion that the process is not capable. Similarly, using either Cp or Cp to assess a process in region 3 would yield the correct decision of a capable process. However, processes that fall in Region 2 have a Cp > CO but a Cp < Co. Those manufacturers and suppliers using Cpx would erroneously reject a capable process. Notice that when -g is greater than 0.1(tolerance), it is impossible to obtain a Cpx of 1.67. Therefore when using Cp, gage error alone can cause a process to be classified as not capable. By Karl D. Majeske and Richard W. Andrews July 17, 2001 Page: 18