Three Years' Experience with Mastery Testing in a Health Informatics Course

Abstract

Background: The necessity created by the pandemic of moving to remote learning presented an opportunity to restructure a multi-departmental graduate-level course with asynchronous mastery testing. With mastery testing, students take module tests when they are ready, and continue to retest on each course module’s core concepts until they perform above a pre-set threshold. In contrast to conventional methods that result in varying levels of achievement, mastery testing enables all students to achieve a high level of performance [1]. Mastery testing has been applied and studied in the fields of mathematics and technology education [1-3], but less has been written on incorporating mastery testing into graduate-level courses in health sciences. A similar method known as the Personalized System of Instruction, or the “Keller Plan”, became very popular in college science instruction in the 1970s [4].

Actions, Methods or Intervention: In the fall semesters of 2020-22, 139 graduate students from the Schools of Public Health and Information, and the medical school’s Department of Learning Health Sciences, enrolled in ‘Evaluation and Research Methods for Health Informatics & Learning Systems’ (LHS660/SI 648/HMP 648), which employed the mastery testing method. The very challenging course material is organized into six modules with mastery tests, where students must achieve greater than 80% in order to complete the module and advance to the next. In parallel with students’ work on the modules, the course included weekly instructor-led synchronous discussions of case studies that apply each module’s concepts. While students progressed through the modules at their own pace, the grading policy included incentives to make a first attempt on each module test by a date that synchronized their progress with the case studies.

Each module has a study guide which contains specific steps for students to achieve the module’s learning objectives and prepare for the mastery test. These steps include textbook readings, recorded lectures, assigned problems, and one-on-one sessions with instructional aides (“coaches”) as needed. The coaches also grade mastery tests using an instructor-prepared template, providing detailed feedback. Students who do not achieve mastery are encouraged to meet with a coach to discuss the questions they answered incorrectly .

The highest grade that a student can receive on a module’s mastery test is determined by which date students take their first attempt. Students can receive an A+ if they take a first test during a ‘Bonus Period’ for that module, an A if the first attempt occurs during the ‘On Time Period,’ and lower grade if the first attempt is made during the ‘Late Period.’

Results: Over three years, the average success rate on mastery tests was 79%, with the trend as: 82% in 2020, 75% in 2021, and 80% in 2022. For all years, 76% of the students achieved success on their first attempts; 20% of the students achieved success on their second attempts; 4% of the students required three or more attempts. In the first year, 2020, there was no incentive to complete a module in a timely fashion once the bonus period ended. As a result, many students fell behind the pace that would keep them in synchronization with the case study sessions. For example, when all students should have attempted a first test on Module 4, only 29% had done so. Students’ critiques on the 2020 course evaluation pointed to a need to “enforce deadlines for the modules because many people in the case study sessions were often not prepared.”

Thus, in 2021, there was a second deadline for each module, creating a “late period”. If a first test attempt occurred during the late period, the maximum attainable module grade was a B. The result was that 78% of Module 4 first attempts were made before the late period, and many students still fell out of synchronization. In 2022, the grading policy was made more stringent, such that first attempts on a mastery test during the late period would lead to a maximum grade of C on the module. The result was that 2% of first tests were attempted in the late period. By the 7th week of the course, 98% of students had attempted a test on Module 4. Students report that this course has a much higher workload compared to their other courses of equal credit and “Preparing for a module test required a SIGNIFICANT amount of work.” Yet, of the 83% of 2021 students who gave feedback, 85% agreed that “This course advanced my understanding of the subject.”

Lessons Learned: The mastery model succeeds in ensuring that all students in the course reach a significant level attainment of the course objectives, which seems particularly important in a course focused on evaluation and research methodology. While the self-pacing that the mastery model affords is appealing in principle, it becomes challenging in practice. With multiple competing demands on students’ time, ensuring steady progress required incentives for keeping pace and strong penalties for not keeping pace.

Future Application and Next Steps: For courses in health education with very challenging objectives, the mastery testing method should be applied with these experiences over three years in mind. The instructor team will continue to iterate on the method based on analysis of student performance and student feedback.