A Preliminary Study of the Impact of Case Specificity on Computer-Based Assessment of Medical Student Clinical Performance

Abstract

The implications of case specificity of two computer-based clinical simulation examination cases (CBX) were examined by a classical measurement approach and by a Bayesian analysis of test characteristics. The CBXs (a surgery and an ob/gyn case) were designed by the National Board of Medical Examiners and administered to 163 University of Michigan Medical School students. The results indicate that the students performed differently on the two cases, the surgery case appearing to be more difficult. The ob/gyn case had greater sensitivity (more accuracy in passing competent students), whereas the surgery case had greater specificity (more accuracy in failing noncompetent students). The differences between the cases and evidence of case specificity raise the issue of an exam's objective and the acceptable type of classification error These results suggest that additional studies are required before widespread use of such exams can be implemented in "high stakes" situations for licensure purposes.