Faculty Evaluation of Student Portfolio Presentations of a Seven-Week Clinical Competency-Based Curriculum Pilot

Abstract

Background: Student portfolios intend to capture qualitative aspects of students' learning experiences, and foster personal responsibility for learning. However, obtaining standardized assessments of portfolios is challenging. This challenge must be met for a true competency-based curriculum in which students have individualized learning paths and educational goals. Summary of Work: During a six-week pilot of a flexible, competency-based curriculum, 5 students summarized their learning experiences for 7 faculty raters who rated each using a novel 5-item instrument. Presentations were 20 minutes long followed by a 10 minute question period. Follow up ratings of videotaped presentations were used to resolve rater disagreements and improve the rating form. Summary of Results: There was low inter-rater reliability of the rating instrument (item intra-class correlations (ICC) ranged from .00 to .91). Follow-up ratings found agreement easier to reach with better-defined item anchors. Conclusions: The difficulties underlying summative assessment of an inherently qualitative experience are likely surmountable. Allowing students to defend their academic progress to a faculty panel in person is enjoyable and worthwhile. Further refinement of a rating instrument will likely overcome interrater reliability issues. Take home messages: Standardized global performance assessment of individualized learning paths is feasible using faculty ratings of semi-structured student presentations.