Extensions of methods for computing sensitivity and specificity for multivariate measurements.

Abstract

Through simulation studies, statistical methods were evaluated and methodological recommendations were made for identifying predictors for obtaining measures of sensitivity and specificity, using multiple decision criteria. Research questions included: (1) whether logistic regression or discriminant analysis should be used for group classification; (2) how to perform cross-validation and consequences of not validating; and (3) how to represent sensitivity and specificity for varied decision criteria (e.g., ROC analysis). Several variable selection methods for statistical models were assessed, while varying stopping rules. One hundred samples of size 300 were simulated by resampling from a data set on 217 Air Force pilots sent to cardiac catheterization. These data (containing measurements on coronary artery disease risk factors) were of interest for the purpose of the identification of factors associated with asymptomatic coronary artery disease. Disease classification was based on percent stenosis. Logistic regression supplied some additional information regarding group discrimination beyond that supplied by discriminant analysis. In variable selection, a more parsimonious model was often obtained using logistic regression (when there was not complete agreement). This model often performed as well as discriminant analysis for measures of discrimination and accuracy. The Score statistic is a reasonable alternative to MLR for variable selection. Increasing entry/stay levels results in decreases in agreement between each of the logistic regression selection methods studied and between each logistic regression selection method compared to the F-statistic method in discriminant analysis. The choice of a selection method for logistic regression is more critical when the entry/stay level is higher. Model validation and tests for goodness-of-fit are necessary. Substantial differences were observed in measures of goodness-of-fit, and discrimination and accuracy between non-validated and validated models. Models may do well with respect to discrimination and accuracy measures even though they do not fit the data. Preliminary analyses using ROC methods on 11 selected samples showed little difference in ROC area between logistic regression and discriminant analysis (i.e., suggesting little difference in discrimination). ROC results demonstrated little shrinkage for each modelling method. The ROC area may be large even if the underlying statistical model does not fit the data.