Understanding Learners and the Interplay Between Metacognitive Judgements of Learning and AI-Generated Explanations

Abstract

Educational tools have the potential to support students learning new material. One of the necessary skills when acquiring new information is metacognition, a subset of self-regulation. This dissertation explores metacognitive judgements, which refers to the process where people monitor their own learning processes and evaluate questions such as how successful they were at retaining certain knowledge. Being able to hone these skills is essential for effective learning. Educational tools can aide this process by presenting explanations to students regarding their responses. Yet, it requires cognitive and metacognitive abilities to make sense of feedback from computer models.

The goal of this dissertation is to better understand the perspectives of learners regarding educational tools and capture their behaviors while in use. By doing so, we discern what and how information can be presented in explanations so that learners are able to make sense of feedback and recommendations generated by the models underlying many educational tools. This can bolster metacognition as learners can incorporate data about what the machine thinks about their learning in order to adapt their learning strategies.

We first understand the broader educational ecosystem using a campus-wide survey about students' views on data use. Acquiring this more nuanced perspective is crucial as trust is an important component for whether or not students adopt and meaningfully use educational tools. We then narrowed the focus by deploying a tool in an online class where students received what they perceived to be AI-generated annotations. This allowed for a clearer understanding of how people leverage a tool to monitor their own learning in authentic settings. Lastly, we conducted a study in a controlled setting where participants had access to a model with tasks that varied in complexity and difficulty. By tracking where participants looked, this yielded a more microscopic view of the specific task and model design conditions that draw or deter people from paying attention.

We find that there are influences on learner's metacognition based on both individual characteristics and how the tool is implemented. Students' personal trust with other authority figures, comfort with technology, and existing knowledge affect their relationship and perceived utility of an educational tool. The transparency of a tool, how well it highlights connections and differences between a student's existing beliefs and the feedback given, as well as the complexity of the tool itself, all serve to impact the its reception and effectiveness. There is an ecosystem of people and choices that must be considered when providing explanations of AI-enabled technology in the classroom.

This work contributes towards an understanding of learners' privacy perceptions regarding educational technologies and how this varies depending on the subpopulation. Such nuances are important to make sure all stakeholders can foster an environment where tools can be effectively adopted and improve learning. We also make contributions in capturing the behaviors and perspectives of students using explanations generated from such systems for the purposes of improving self judgements. Specifically, drawing novel ties between Explainable AI (XAI) in educational settings to literature on learner feedback and Open Learner Models (OLMs) through field and experimental investigations. We develop a fuller understanding between metacognition and individual differences incorporating demographic backgrounds and technology trust levels, allowing models to be presented in a way that is digestible to learners and enable them to better reflect on their own knowledge.