Improving and Assessing Propensity Score Based Causal Inferences in Multilevel and Nonlinear Settings.

Abstract

Recent calls for accountability have focused on scientifically based research that isolates causal mechanisms to inform both the policies and practices of education. A major challenge in aligning educational research with such standards has been to develop methods that can address the interdependency and multilevel structure of teaching and learning and approximate randomized experiments using observational data. In this dissertation, I carried out three studies that centered on improving causal inferences drawn from observational studies in common educational settings. In the first study, I developed several models for estimating multilevel propensity scores (PSs) and examined their effectiveness for causal inference. The results suggested consistent gains from multilevel PSs that allow differential influence of the group on its individuals. The results further suggested that covariate selection in multilevel PSs can play a large role, both relative to model type and in an absolute sense. The second study then developed a method to construct PSs in an effective and efficient manner using two pivotal relationships. The method made use of each covariate’s relationship with the treatment and commonly available outcome proxies (e.g. pretest measures) to construct PSs that minimizes the mean-square error (MSE) of the treatment effect estimator. The results of the study suggested that an effective and efficient approach to constructing the PS might be to include those covariates whose relationship with the outcome is at least half the magnitude of the respective relationship with the treatment. In the final study, I develop an index that assesses the sensitivity of inferences in binomial regression models by extending the impact threshold of a confounding variable framework (Frank, 2000). Each of these methods is then applied to observational data to demonstrate how these methods can advance the quality and robustness of causal inferences in educational research.