What Do We Get Out of College? Measuring and Teaching Critical Thinking Skills

Abstract

Many employers state that they expect college graduates to have “soft” or domain-general skills like critical thinking, and universities claim to teach these skills. To prove colleges are delivering on their promise—and to demonstrate that employers are picking the best personnel fairly—colleges, students, and employers need a valid measure for critical thinking. This dissertation proposes a highly ecologically valid measure and uses it to answer key questions about the added value of a college education, specifically for the development of critical thinking. Different stakeholders have different ways of defining critical thinking. Older definitions are based around logic, evaluation, and reasoning. In the digital age, however, a conceptualization of critical thinking that respects the importance of source analysis skills is paramount. I propose a multifaceted definition of critical thinking that includes deliberate and logical decision-making, but also includes source analysis, balanced argumentation that avoids myside bias, and clear communication of the deliberative process. Seminal work in the field of critical thinking has estimated that college can account for between 0.5 and 1.0 standard deviations of critical thinking development. I discuss the different ways researchers have measured critical thinking to come to this conclusion in a literature review before moving on to the proposed measure of the dissertation. I developed multiple, ecologically valid measurements as performance assessment tasks. Performance assessment tasks are curated, authentic tasks that reflect the real-world use of a complex skill. In my evolved version of performance assessment tasks, critical thinking skills are evidenced by (1) prolific and balanced argumentation, (2) source analysis regarding trustworthiness and relevance, and (3) quality of written communication. I calibrate two different tasks (“Bryn Bower series”) with each other to minimize retesting effect in longitudinal studies. In addition to presenting the results of this calibration study, I demonstrate in a series of studies the reliability and validity of the Bryn Bower series based on the newly developed count-based coding scheme. To answer our substantive research questions about the impact of college education on the development of critical thinking, I demonstrate a lack of cross-sectional differences between underclassmen and upperclassmen and a general a lack of development longitudinally, confirming the cross-sectional results. Finally, I analyze the development of critical thinking of students in an intense critical thinking seminar that includes a debate activity in which the students practice critical thinking and compare them to student sample with only partial intervention exposure. Students develop their critical thinking more with more access to the intervention. Altogether, the findings from these studies call into question the received wisdom that college improves critical thinking. More specifically, our highly ecologically valid measure does not indicate that the suggested range of effect sizes in the literature is reasonable. However, with the explicit instruction and opportunities for guided practice as exemplified in our intervention, these estimates effect are within reach. I discuss the implications of our research findings for higher education in the final chapter.