Artifacts and Hidden Processes: Veracity Does Not Guarantee Validity in Judgment-Based Measurements

Abstract

This dissertation explores the critical issue of measurement validity in factual judgment- based assessments within mass communication and media studies. While factual judgments are often assumed to provide valid and objective measures due to their verifiable nature, this dissertation challenges that assumption by demonstrating that veracity does not necessarily equate to validity using two familiar examples, the Cognitive Reflection Test and the first-order cultivation indicators. The dissertation uncovers how various measurement artifacts can distort judgment-based assessments, leading to systematic biases and hidden processes that have been underexplored in the literature. The dissertation is structured around two primary objectives based on two empirical projects. First, it investigates specific instances where factual judgment-based measures, though seemingly verifiable, fail to capture the underlying constructs they are intended to measure based on veracity. Through the reanalysis of existing data and the application of alternative question- wording and validation criteria, the research reveals that these measures often reflect artifacts and hidden processes rather than the constructs of interest. For instance, the Cognitive Reflection Test assumes that the two primary answer categories map well onto the two distinct mental processes. In contrast, the commonly assumed override of intuition may not occur when participants solve the problem. Among some populations, the assumed intuitive answers of CRT indicate ample cognitive effort and adequate performance. Second, the dissertation critically assesses the broader theoretical implications of relying on potentially flawed measurements. It argues that the uncritical adoption of these judgment- based measures has contributed to a significant conceptual-empirical gap in the field, where theoretical constructs are not adequately reflected in empirical research. For instance, in the context of cultivation theory, which suggests that heavy television viewing leads to a distorted worldview, the dissertation demonstrates how measurement artifacts can undermine the validity of such claims, suggesting that the observed effects may be, at least in part, artifacts of the measurement process rather than true media influences. Collectively, people are accurate in estimating gender ratios of populations, and TV exposure may facilitate more accurate estimates. The dissertation demonstrates that more rigorous validation practices in developing and applying factual judgment-based measures are fruitful and crucial to existing theory development and cumulative research. This includes triangulation methods, and alternative measurement approaches that account for the hidden processes identified. By critiquing the rigor of conventionally used measures, this dissertation underscores the urgent need to reevaluate current research practices to enhance the validity of findings in communication studies and provide a more solid foundation for future theoretical development.