Design and Pilot of a New Clinical Evaluation Tool for Student Registered Nurse Anesthetists

Abstract

Purpose: Observed student clinical performance does not always match scores on the currently utilized clinical evaluation tool at the University of Kansas Medical Center (KUMC), and faculty members do not believe the current tool is sufficiently evaluating student clinical performance. A review of student and faculty satisfaction surveys from previous years showed a consistent dissatisfaction with the clinical evaluation tool utilized by the nurse anesthesia education department. This has also been a trend nationally amongst nurse anesthesia education programs (NAEPs). The purpose of this project was to design and pilot a new clinical evaluation tool used to evaluate clinical performance of student registered nurse anesthetists (SRNAs) that was reliable and valid. Faculty satisfaction with the new tool compared to the currently utilized clinical evaluation tool was measured as a secondary outcome. Methods: A literature search and focus group of nurse anesthesia education faculty was conducted to help determine criteria for a new clinical evaluation tool. Using that data, a pilot clinical evaluation tool was developed. Expert review of the pilot tool was completed by certified registered nurse anesthetist (CRNA) faculty at the pilot facility as well as outside facilities. Following expert review, necessary changes were made to the pilot tool, and the pilot tool was trialed for 4months. The pilot tool was administered in addition to the current clinical evaluation tool to evaluate SRNAs in their first clinical semester. At the end of the 4 month trial period, the pilot clinical evaluation tool was analyzed for reliability and validity. Scores from the pilot tool were evaluated and compared with scores from the current evaluation tool. Another focus group was held with KUMC faculty to gather feedback, assess effectiveness, and evaluate satisfaction with the new tool. Results: The pilot clinical evaluation tool was used to evaluate a total of 24 novice-level SRNAs during a 4 month pilot period. A total of 119 pilot evaluations were completed. The pilot clinical evaluation tool was tested for content validity, parallel forms reliability, and interrater reliability. Item-level content validity index (I-CVI) scores were between 0.75 and 1 for relevancy and clarity. Scale-level CVI (S-CVI) was rated 0.97 on relevancy and 0.69 on clarity using the universal agreement method of computation. Using the less conservative average method of S-CVI, the pilot tool S-CVI was rated 0.99 on relevancy and 0.92 on clarity. A Pearson product-moment correlation was computed to determine parallel forms reliability. A correlation of 0.717, p < 0.01 was achieved. There were 68 possible agreements and 37 actual agreements and an interrater reliability coefficient of 0.544. Conclusion: The pilot clinical evaluation tool was found to be valid and reliable. Scores between the pilot tool and the current tool were well-correlated. Faculty satisfaction with the pilot clinical evaluation tool was greater than with the current tool. There was room for improvement in the clarity of answer choices on the pilot clinical evaluation tool.